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ABSTRACT 

Criteria by which an organization should select a 
vendor-supplied training program and the measurei«ent and rating of 
progreuns against such selection criteria are described. A review of 
the pertinent literature and research, from a theoretical rather than 
practical perspective, is included. Factors and criteria identified 
in the literature as important to evaluating a vendor-supplied 
progrcun can be clustered around four dimensions: (1) content factors 
and their fit to the needs of the organization; (2) design factors 
and their utilization in the program; (3) control factors and the 
research and evaluation supporting the program; and (4) context 
factors and the fit of this program into the administrative aspects 
of the purchasing organization. A checklist of the identified 
criteria in thes^ areas was created and used to develop an instrument 
with worksheets to compare the program under consideration to the 
model. Use of an instrument of this type would help ensure that 
progreuns were purchased on their merits and that flaws would be more 
readily apparent. Costs of training might drop and the importance of 
training would be enhanced in the organization. A 134-item list of 
references is included. Forty-seven figures illustrate process 
flowcharts and the worksheets for program evaluation. (SLD) 
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CHAPTER ONE - INTRODUCTION 
When organizations use outside sources to meet their training 
requirements, managers and training professionals are required to 
make intelligent, rational, and informed decisions about which 
training program would work best in their situations. Yet, processes 
and guidelines that can be used to assist people in making such 
decisions have not been well defined. If this decision-making 
process could be improved, a notable increase in return on 
Investment for training professionals would result, and the role of 
training program buyers would be enhanced. This investigation will 
study the criteria and methodology used when managers/trainers 
decide which vendor-supplied training prog.'ams to evaluate and 
select. 

The process of evaluating training programs receives 
increased emphasis as training in a corporation becomes more 
closely aligned to the organization's strategic plans. A review of 
the literature shows that the emphasis in evaluation research has 
been placed on the factors involved in evaluating training after it 
has been delivered. For example, Kirkpatrick's four-step evaluation 
hierarchy of reaction, learning, behavior, and results reflects an 
outcome orientation for evaluating training (1975, 1976, 1983). The 
focus of the previous research on training effectiveness and 
outcomes and the more recent emphasis on cost-benefit analysis 
(see Swanson & Gradous, 1988) represent critical elements in 
substantiating the legitimacy of the emerging training and 
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development field and the credentials of the tiaining and 
development professional. Although much has been accomplished, 
the history of evaluation in trailing suggests that considerably more 
effort is required to improve the decision-making processes used in 
selectir.g, designing, developing, implementing, and evaluating 
♦raining programs. 

Goldstein noted that despite important recent advances in the 
development of evaluation models, "the most serious problem has 
been the failure even to consider examining the instructional 
methods" (1986, pp. 112-113). Brinkerhoff (1987) further stated: 

There are many reasons to be concerned with evaluating HRD 
programs as they happen, well before they have had a chance to 
produce results. In fact, to look for effects only after the 
program is to perpetuate trial-and-error learning. If 
evaluation during the early developmental stages can show 
that a program is ill conceived or poorly executed, then there 
may be a good reason to revise or even abort it. Evaluation 
made a part of the program development process can help 
programs succeed, as well as measure whether or not they do. 
(p. 66) 

Purposes of the Study 
This study considers selection criteria for choosing a vendor- 
supplied training program and describes how to measure and rate 
programs against identified selection criteria. A review of the 
pertinent literature and research is included. The identified criteria 
are organized into an evaluation model and instrument that can be 
used to guide the buyer in selecting vendor-supplied training 
programs. 



The methodology by which this investigation was accomplished 
includes: (a) reviewing the literature, (b) identifying selection 
criteria for evaluating vendor-supplied training programs, (c) 
creating a model for the selection of vendor-supplied training 
programs, (d) creating an instrument and methods for gathering 
information based on the model, and (9) making conclusions and 
recommendations for further investigation. The outcomes of this 
study are a model and an instrument that can be used when 
managers/trainers compare alternative training programs before 
purchasing a pilot program. 

Definition of Terms 
Because a variety of terms have been used by various 
individuals and organizations to describe certain elements in the 
field of training, the subsequent definitions will serve to clarify the 
terminology used for this investigation. 

Program refers to a course, package, class, or similar item 
related to the training of individuals in organizations. 
Vendor refeis to any source external to the purchasing 
department or company that offers a program for use in 
training personnel. Vendor refers to the company or individual 
who offers the program to the purchasing organization in 
exchange for money. The exchange of money does not affect 
the model or this investigation, other than to serve as a 
criterion to be considered. Thus, the word vendor will be used 
synonymously with phrases such as external source, supplier, 
and outside provider. 
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External sournft could include sources other than vendors, and 
it refers to the source of any program purchased for the 
purpose of training, regardless of where the training will be 
held or who will provide the training. For example, the source 
of the prog.-am might be another division of the same 
organization. In this investigation, the focus is on evaluating 
programs that have been developed outside the purchasing 
organization. In this paper, it makes no difference where the 
program originated or the terms of the exchange, although the 
decision maker may consider both the program originator and 
the terms of the exchange as important criteria. 
Criterion refers to a measure against which bome aspect of a 
program might be evaluated. 

Dimension refers to a set of criteria that have some element 
in common. For example, the "content dimension" would 
consist of a number of criteria that could be considered in 
evaluating a program's content. 

Purchasing organization refers to the organization that will 
use the program. It may refer to a department, a company, a 
division, a customer, or an individual, depending on the scope 
of the intended use of the program under consideration. 
Decision maker refers to the person (or group of individuals) 
who maKes the decision or recommendation concerning the 
training need in question. The decision maker could be the 
training manager, an experienced designer, an instructor of 



training programs, a line manager, or any concerned person (or 
group of people) involved in the evaluation process. 

Statement of the Problem 
The use of vendor-supplied programs in training is increasing. 
Training magazine's 1987 industry report revealed that the dollars 
spent on vendor-supplied services and materials increased by 12.2 
percent during the preceding year. Almost $7 billion was budgeted 
by training departments for external purchases for 1987 (Lee, 
1987a). Over $4 billion of that $7 billion was budgeted for off-the- 
shelf materials, seminars, conferences, and customized materials 
(Geber. 1987). 

A study in the May 1985 issu« of Performanre and Instmrtinn 
Journal reported that 42 percent of the companies surveyed (n - 88) 
developed less than 25 percent of their training materiaL and 
programs in-house. Another 14 percent developed less than half of 
their training in-house (Schoonmai^er, 1985). Based on this study, 
over one-half of the training provided by these companies originated 
from external sources. Although the validity of the study may be 
open to question, the overall results indicate increased use of 
external sources in the training industry. 

Thus, studying the decision-malting process by which managers 
select external training programs is important. An increased 
understanding of the criteria used to evaluate and compare external 
programs must be explored in orde,- for managers to make wise and 
effective purchasing decisions. Practical criteria by which to 
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evaijate and compare the programs can be gathered and formatted 
into a useful selection model for decision makers in organizations. 

Research has been and is being conducted on both the 
evaluation processes used to determine a training program's 
effectiveness and on a m'^thod to develop cost-benefit analysis of 
training programs. This study will nsii focus on these areas of 
training evaluation; instead, it will focus on the criteria that are 
critical for making an informed and wise selection of pre-existing 
and customized programs. Given a number of programs Inat could 
potentially meet a set of iaentified training needs, a method for 
evaluating each program before it is piloted or purchased will be 
identified. 

Significance of the Problem 

Increasingly, training managers are being held responsible for 
decisions that affect the bottom-line results of their organizations. 
The effectiveness of the training function may provide the 
difference between success and failure for certain positions and 
responsibilities. For example, as markets become more competitive, 
well-trained sales people can make the critical difference between 
obtaining, holding, or losing an account. As work technology 
changes, training people efficiently and effectively in specific, new 
skills may be a critical element in a company's ability to adapt 
strategically to and compete in a competitive marketplace. 

Decisions to use a vendor's program can include a substantial 
commitment on the part of the buyer in terms of money, time, 
personnel, and professional credibility. Making better decisions 
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concerning the purchases of vendor-supplied training programs could 
result in increased benefits for the organization and the decision- 
maker. Similar to other capital expenditures, training represents an 
investment in the organization's future. To maximize the return on 
tho training investment, the decision maker needs to consider each 
required aspect o* the vendor's program. 

As the strategic importance of training increases, the 
decisions of training managers are being scrutinized more carefully 
by others in the organization. As training expenditures increase, top 
management may require more justification and analysis before 
approving training expenditures. Therefore, a format to aid training 
buyers in making better selections and justifying their decisions to 
purchase or to use pilot programs can be helpful. 

An examination of the decision-making process could provide a 
systematic methodology by which to make such decisions. An 
individual trainer may make a series of one-time purchasing 
decisions, but by creating and substantiating a selection model, a 
system can be set up to guide future decisions. For a buyer, having a 
system could lead to better decisions and the outlay of less time and 
energy. Such a system could be personalized to fit the unique needs 
of the organization, and improvements could be incorporated into it, 
increasing its effectiveness over time. 

The results of this investigation could be of interest to 
training vendors, buyers, managers concerned about the value of 
vendor-supplied training programs, and training and development 
researchers. The selection model could provide a coi.iprehensive set 
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of criteria for decision makers to use when evaluating vendor- 
supplied training programs, thereby improving their selection 
process. Internal developers of training programs could benefit 
from using the methodology as an evaluation tool. Vendors could use 
the selection model to evaluate their programs before offering them 
on the market, and their marketing approaches could be changed to 
provide the information the buyer requires to make an informed 
decision. Managers can use the system to arrive at a more rational 
decision in making and approving training purchases. The trainer can 
use the selection model to provide the background for his or her 
recommendations to management. 

The significance of this study is also affected by managers' 
perceived purposes of the evaluation process. Certainly, one purpose 
of evaluation is to help the manager make go/no-go decisions, such 
as whether or not to continue a particular program, to buy inside or 
outside, or to develop new programs. Another purpose of evaluation 
is to provide data concerning the ongoing effectiveness and results 
of the program. The program evaluation selection process that is 
the focus of this investigation could also provide the data required 
to improve existing programs, whether they are developed internally 
or externally. Besides providing the information required for a 
go/no-go decision, the instrument and model may represent a valid 
process for identifying specifically what is required to adapt or 
customize an outside vendor's product for the buyer's unique 
requirements. By using this method of evaluation, the decision 
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maker may uncover previously unknown concerns and needs, and 
these could be incorporated into the selection process. 

Major Research Questions 

In this study, it is assumed that the criteria by which 
programs are compared can be identified and that these criteria can 
be organized to provide a method for systematically scoring 
alternative programs; this permits the buyer to make a more 
informed, more intelligent, and less biased decision. Although it is 
acknowledged that decisions to purchase training programs do not 
necessarily follow a rational process, it is assumed that using a 
more rational process will result in a more effective decision. 

The following major research questions will be investigated 
as a means of pursuing the stated problem: (a) What selection 
criteria can be used to evaluate external training programs before 
they are pilot-tested or implemented? (b) What effective, efficient, 
and objective methods have been identified for measuring each of 
the above criteria? and (c) Can these criteria be organized logically 
into a useful selection model and instrument? 
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CHAPTER TWO - REVIEW OF THE LITERATURE 
Little has been written about the decision-making process 
used in selecting vendor-supplied training programs. Of the 
literature that exists, none is based on quantitative research. More 
has been written about the process used in developing programs. The 
literature on program development and evaluation may serve as the 
basis for identifying factors to use in evaluating and selecting pre- 
existing programs. Hero, issues associated with various program 
components, the evaluation of training programs, the development of 
in-house programs, the purchase of outside programs, and the 
decisior^-making process are conridered. Finally, covered in this 
review is information concerning where the decision maker can find 
out about vendor-supplied training programs. 

The reasons an organization can have for using an outside 
source to fill a training need and the criteria by which an 
organization selects a program are not mutually exclusive. For this 
study, a review of the literature concerning reasons an organization 
woi'ld have for purchasing training outside versus developing it in- 
house v/ill be considered first, and the literature concerning 
selection criteria will be considered second. 

Build In-House or Purchase From An Outside Source 
Changes in the perceptions of organization, work, and workers 
have resulted in comparable changes in the focus of the training 
function within organizations. For example, as quality has become 
an issue of strategic importance to businesses, the importance of 
training employees in the use of statistical process control systems 
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has increased. Some of these changes have resulted in companies' 
relying on external sources for program development. What factors 
influence the decision to develop a program in-house as opposed to 
obtaining it from an outside source? 

An organization's decii?ion to purchase a program from an 
outside source is rarely obvious (Cothran, 1987). Rather, as in the 
evaluation process in general, decisions emerge out of a process 
that includes various individuals' input, such program factors as 
cost and availability, and other organizational considerations. 

Benefits associated with designing a program in-house include 
a greater amount of control over factors such as quality, content, 
scheduling, materials, media, selection of instructors, and the 
learning perspective taken in the program. The development process 
may provide opportunities for individuals from different 
departments and disciplines to interact, resulting in better working 
relationships and facilitating learning on the issues involved in the 
program (Kirkpatrick, 1985b). The applicability of the program to a 
particular organization can be developed from the beginning of the 
process. Cultural values that are important to the organization and 
assumptions concerning previous learning experiences of the 
trainees can be incorporated into the program development p'ocess, 
starting with the program's conception. 

On the other hand, external programs may offer several 
benefits that in-house packages do not. Four hundred and thirty- 
three respondents to a survey of business people on what criteria 
are important to consider when deciding to buy training outside 
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rather than to develop it inside revealed that half of them rated the 
range of resources available as the most important criterion. Other 
criteria they selected as important included quality of the end 
product (43%), speed of delivery (34%), cost (34%), capability of 
outside vendors' personnel (33%), and unique technology (27%) 
(Lashbrook, 1981a, 1981b). Despite limitations of Lashbrook's study 
regarding the variety and selection of subjects and a possible bias 
toward the vendor's perspective, this investigation considers the 
results of her study in greater depth. 

Organizational Capability 

A range of resources, capability of training personnel, and 
unique technology can be combined into an overall factor concerning 
the organizational capability of the vendor. This, then, is the main 
reason for the decision to buy outside (Lashbrook, 1981a). Some 
training programs may require greater in-depth knowledge, skills, or 
technology than are available within the purchasing organization. 
Lacking internal resources, the decision maker must choose an 
outside source that has the expertise to develop the program. 

Certain types of training and certain issues may be better 
presented and designed by outside vendors. If a training issue could 
be met with a great amount of resistance by the participants, thus, 
representing a high political risk for the training department, an 
outside source may provide a safe alternative to minimize the 
potential negative political outcomes. The "Cassandra" effect may 
be operative; that is, internal prophets will not be believed despite 
their qualifications, yet an external source could provide the 
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credence required to make the training successful (Cantwe.'., 
Hosterman, & Sholton. 1976). 

When technical expertise exists within the organization, there 
could be a tendency to decide prematurely to develop the training in- 
house, without adequate consideration of other factors. Technical 
expertise is only one aspect of the decision maker's choice. 

No firm decision regarding the type, format, or source of 
training should be made unless sufficient attention has been 
given first to identification of the trainee population and the 
establishment of Instructional objectives; and even with this 
attention, the decision will be obscured by factors which may 
have little to do with training. The trainer needs to consider 
each of these factors carefully. (Cantwell et al., 1976. p. 47-5) 

Of course, the proprietary nature of tightly controlled or sensitive 
Information may be an important factor in some instances (Cotfiran, 
1987). 

Custom nfigj^nir^q 

According to Lashbrook (1981a), the single most important 
factor in choosing an outside source was the vender's ability to 
customize programs (20% of the respondents). This adds to the 
importance of the overall capability of the vendor. Buyers want a 
program that is adapted to the unique needs of their organization, 
and they will often consider the vendor's capacity to customize the 
program the most critical element in selecting a vendor. 

Abilitv to Dftlivftr 

The quality of the end product and speed of delivery can be 
considered as the external resource's ability to deliver. Since these 
were rated as the second and third most important criteria 
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considered when buying an outside program, the vendor's ability to 
deliver is also a very important consideration (Lashbrool^, 1981a). 
Because quality was rated higher than cost, it appears that 
organizations are clearly interested in purchasing programs that 
accomplish their objectives rather than purchasing programs solely 
according to their cost. Quality was more important to respondents 
who were actual decision makers or who spent a higher proportion of 
their budget on outside sources (Lashbrook, 1981a). 

Time constraints are another important consideration in 
choosing an external program. Larger companies with larger training 
budgets are more likely to be concerned about delivery than are 
other buyers. The ability to provide the training at the most cost 
effective time when it will produce maximum results for the 
organization could result in greater overall cost savings, even 
though the program itself may be more expensive. Thus, timing may 
be the primary decision factor. In order to build a program 
internally, training personnel must research the program, develop it, 
evaluate a pilot, and, if necessary, revise it. Because of time 
constraints, training personnel may use external sources to meet 
their internal timing needs. 

Cost 

It might be assumed that cost is the most important reason 
why decision makers turn to external sources for training programs. 
Yet cost ranked fourth in this survey. The people actually 
responsible for making a buying decision rated cost higher than did 
the people involved in a less direct manner in the buying process. 
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Large companies were more concerned with cost than were small 
companies. The real issue with cost is probably cost effectiveness 
(Lashbrook, 1981a). Other factors, such as the number of employees 
being trained and the projected work load of the internal design 
team, may raise or lower the importance of the cost factor. 

Cost considerations are rarely of the nature of simply buying 
the best program. Considerations such as the vendor, the content, 
the potential fit within the organization, the possibility of 
customization, and budget limitations must be balanced against the 
cost to achieve the best choice requiring the least money. Training 
program buyers should be prepared to argue that one particular 
program is the best choice, not because it is the least expensive, but 
because it is the most cost effective program, given the many 
factors affecting the program's success (Brinkerhoff, 1987). 

Other Considerations 

Because of the unpredictability of their market, many 
organizations use external resources, such as independent 
contractors and consultants, to maintain more flexibility with their 
interr.al work force. Thus, the people power required to meet a 
newly defined training need may not exist within the organization. 
Therefore, an organization that lacks the training personnel needed 
to develop a new training program may use an external source rather 
than hire new employees. 

Summary 

Sredl and Rothwell (1987) identified several criteria that 
could influence the "build or buy" decision. They asked a question 
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concerning when alternatives to formal, in-house instruction ^f-ould 
be considered, and offered the following considerations: 

1. Purpose. Is the experience largely intended to keep 
learners abreast of changes outside the organization? If 
so, external offerings are most suitable because learners 
will encounter many different people from other 
organizations. 

2. Specificity of treatment. Is the experience intended to 
be tied to the unique policies, procedures, and methods of 
the organization? If so, external offerings will not be 
appropriate because they rarely take such specifics into 
account. 

3. Size of the group affenteri Is the need limited to one 
person or a few people? If so, alternatives to in-house, 
formal training should be considered. In most cases, 
formal in-house training should focus on recurring 
needs-like the job orientation of new employees. 

4. Available expertise. Is necessary expertise available in- 
house? If not, alternative sources of instruction should 
be considered. 

5. Importance of the w ork group . How important is it that 
the learning experience take into account the unique 
norms and status hierarchy of one or more work groups in 
the organization? 

6. Cost-henefit ratio. Is it more cost beneficial to send 
people out, design in-house instruction, or use some 
other method to meet a need? 

7. Policies. Is a subject for instruction too sensitive or 
potentially explosive to be handled effectively in-house? 
If so, alternatives to in-house instruction might be more 
appropriate. 

8. Necessary integratio n with other experience . Is it 
important that a learning experience fit precisely into a 
larger structure or sequence of related programs, most 
of which were designed in-house? (pp. 92-93) 

These considerations combined with the issues concerning the 

vendor's capability and resources, expertise, cost, and delivery time 

frame provide a comprehensive list of the factors found in a 
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decision to use an outside vendor rather than to develop an in-house 
program. 

Selecting Vendor Training Programs 
Several sources directly concerned with the selection of 
vendor-supplied training programs were identified. Each of the 
sources is reviewed, a summary is made of the common features, 
and comments are made concerning the differences. The initial 
focus is on the criteria that are important tc selecting a program, 
not the criteria used in the decision on whether or not to use an 
outside supplier. 

Rogers and Voipe (1984) created a five-level, no-nonsense, 
common-sense model that considered the following: 

Level 1: Weeding nut iunk mail Quickly assess the value of 
promotional literature using the' following standards: 

a. Are there any stated learning objectives? 

b. Is the training performance-based (that is, does it 
promote skills rather than attitudes)? 

c. Is there a topical outline of the program? 

d. Does it seem to have more substance ihan simply a 
smart marketing attempt? 

Level 2: Taking a ninsflr inr^^; Obtain the following from the 
firm selling the package: 

a. Detailed content outline 

b. Participant materials 

c. Sample of program 

d. Instructor information 

e. Testimonials from other companies 

f. Cost breakdown 

Level 3: Building a comparison matriy Compare the specifics 
of the program to your actual training needs to see how the 
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two match up. Compile a simple matrix, in which your training 
needs are compared against the topics that the program claims 
to address. 

Level 4: Instruction al desipn analysis . This level involves a 
detailed instructional design analysis. It is here that you 
approach the program from an instructional design standpoint. 
Using a top-notch instructional designer on staff or an outside 
consultant, you would touch on the following design issues: 

a Do objectives and content match with needs 
analysis? 

b. Is there internal coi.oistency among instructional 
objectives, criterion tests, and learning 
activities? 

c. Are prerequisite skills identified? Are they 
congruent with the target population? 

d. Is the sequence of instruction based on Gagne's 
learning hierarchies? 

e. Is the instructional strategy appropriate? 

f. Are complete and adequate instructional activities 
provided for each lesson? 

Level 5: Running the pilot. Evaluate the program while the 
pilot is in session based on the specific instruments you have 
made alohg the way. (pp. 18-21) 

Emphasizing the organizational factors over the program 
characteristics resulted in the following eight-step model by 
McDevitt (1983): 

1. Determine relevant organizational factors. 

Look at: Objectives 

Authority structure and style 
Climate 

Training history 

2. Examine vour training objectives . 
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Are your objectives congruent with organization's 
objectives? 

Look at: Nature of training problem 
Trainee characteristics 
Potential benefits to organization 

3. Examine v/ftndor progr am Qb|ectivftfi 

Are they clear and defined in behavioral terms? 
Can they be measured? 

Do they match closely your training objectives? 

4. Contact other vendor p rogram nsflrs 

Look at evaluations from comparable organizations 

5. Research vendor program 

Contact local business schools 
Examine written reviews 

6. Examine methodology 

Look at: Methodology of the program 

Methodology that has been successful for your 
organization 

7. Decide on vendor program 

Choose from alternatives 

Have decision reviewed by colleagues 

Sample the program, if possible 

8. Do Dost-program evalu ation and share the results . 

Write review or document evaluation results for others 
(p. 83) 

Cantwell, Hosterman, and Shelton (1976) discuss two key 
issues associated with the selection decision: competence in 
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subject matter and instructional design. Both issues must be 
addressed in the selection process. The rationale for valid subject 
matter in a training program is critical, but no less important is the 
design and execution of the program. 

The prospective external resource should be able to document 
specific instructional objectives, a rationale for how the 
instructional activity relates to the objectives, and criterion 
tests which will indicate whether the student can perform as 
intended. (Cantwell et al., 1976, p. 47) 

Each of these elements should be considered as selection criteria. 

A different perspective on selecting and evaluating vendor 
supplied training programs was presented in McAllaster's doctoral 
thesis (1987). He assessed vendor-produced, off-the-shelf 
programs according to how they conformed to the dimensions of: 
adult learning principles, program development models, and teaching 
methods. He used a case-study method and interviewed vendors 
from three companies, program participants, and instructors. His 
conclusions showed that each of the three programs reflected an 
awareness of selected adult learning principles, and a program 
development model was considered when they were djsigned. 
However, the programs required modifications based on program 
design, development, and instruction before they could be 
implemented by the purch^ding organization. 

In terms of teaching methods, a potential problem existed 
between what the vendors claimed and what the actual requirements 
were for implementing the training in a company setting. The 
vendors' literature and sales efforts indicated that almost anyone in 
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the company could be an instructor. However, the trainer-training 
did not provide or address the prerequisite skills required for an 
instructor: the ability to (a) understand adult learners, (b) modify 
programs, and (c) evaluate them. In effect, the instructor was the 
critical element in making the generic materials and activities of 
the program relevant to the company and the participants. The 
effectiveness of the training was highly dependent on the 
instructor's skill and abilities. 

In addition, McAllaster considered how the programs were used 
within a company setting and what effect organizational factors had 
on each program's effectiveness. His results indicate that 
management support for the training was the most important 
organizational element for ensuring the training program's success. 

Without active management support throughout the 
organization any training program is in for a difficult, if not 
imDossible, time. Just paying vendors' L/olces, providing 
time away from wcrk, lost productivity, and travel expenses 
does not guarantee that a company will benefit from a 
management development program. (McAllaster, 1987, p. 266) 

In order for it to be effective, the training must be related to the 

business, the culture, and the environment of the organization. 

In conclusion, a set of guidelines for assessing vendor 

programs was given. In this case study, the purchasing company 

failed to assess clearly what had to be accomplished with the 

vendor program before the decision wjS made. Thus, the first set of 

guidelines addressed the following factors associated with the 

program's purpose and organizational fit: 

A. Assessment of Internal Requirements 
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1. Formulation of the ProDlem 

2. Identification of the Program Goals 

3. Determination of Goal Congruency 

4. Determination of Management Support 

5. Identification of Internal Resources 
{McAllaster,1987, p. 271) 

The second set of guidelines assessed the vendor's program on the 

basis of the following factors: 

B. Reviewing Vendor Programs 

1. Program Goals 

2. Program Material and Activities 

3. Instructor's Role 

4. Evaluation Methodologies 

5. Client Support Services 

6. Follow-Up Programming Possibilities 
(McAllaster, 1987, p. 271) 

Together these guidelines are intended as a complete list of factors 

to consider when buying a vendor program. Whether using these 

guidelines would result in more effective purchasing decisions was 

not researched in this study. 

Information regarding thf> selection criteria identified by 
Lashbrook {1981a) in Training magazine formed the basis of Jack 
Phillips' (1983) discussion of criteria to consider in evaluating 
outside resources. Essentially, the criteria identified in the study 
as influencing how and why organizations use outside sources are 
also used as the criteria by which to evaluate the programs. 

Tables 1-3 summarize the results of the previously mentioned 
researcii concerning vendor-supplied program selection. In effect, 
cost, quality, ability to deliver, and capability considerations are 
the main factors influencing an organization's decision to purchase 
outside programs. Cost and ability to customize programs are the 
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key aspects in selecting an outside vendor. Cost, the need to solve 
difficult problems, and obtaining new perspectives and knowledge 
not available internally are important advantages in using an outside 
source vendor. In addition, the Human Resource Development (HRD) 
department was assessed to be in the best position to evaluate the 



Table 1 

Criteria for Buvlno Outsic'e Rather T han Daveloplno Insida 
Why Buy Outside Rather than Develop Inside?* 



Range of resources available .49% 

Quality of end product .43% 

Speed of delivery 34% 

Cost .".'340/, 

Capability of personnel 33% 

Unique technology 97% 

Other 1 30^ 



'Since respondents could select more than one factor, the total exceeds 100%. 
(Taken from a survey conducted by Training magazine and the Instructional Systems 
Association.) 
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Table 2 

Criteria for Deciding Among Outside Vendors 

When Selecting an Outside Training Vendor, What Are the Most Important 
Factors In Your Company Decision?* 



Ability to customize programs 64% 

Cost 60% 

Favorable experience with vendor 56% 

Reputation 55% 

References/recommendations 4 9 % 

Capability of instructor 40% 

Research/development capability .33% 

Depth of product line 20% 

Media compatibility 11% 

Influence of the salesperson 4% 



*Since respondents could select more than one factor, the total exceeus 100%. 
(Taken from a survey conducted by Training magazine and the Instructional Systems 
Association.) 



Table 3 

Attitudes Toward Outside Vendors 



Outside Vendors Agree* 

Bring fresh perspectives 83% 

Help solve difficult problems 61% 

Spread costs over clients 54% 

Malie practical applications 51% 

Tailor programs 50% 

Tend to be leaders in software and conceptual material 47% 

Promise more than they deliver 41 % 

Rip you off 1 1 % 



*Sir)ce respondents could select more than one factor^ the total exceeds 100%. 
(Taken from a survey conducted by Training magazine and the Instructional Systems 
Association.) 
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effectiveness of a program and to make the buy/build decision 
(Lashbrook, 1981a; Phillips, 1983). 

In considering the selection of the best program from several 
alternatives, Phillips suggested using a three-point comparison 
ranking, weighting each criterion, and then solving to determine the 
t St choice. If possible, pilot test the program before purchasing it. 
A written guarantee should be secured from the vendor. Even though 
the organizational factors involved in the implementation of a 
program are important, a vendor who has confidence in the product 
should be willing to guarantee the program's effectiveness 
(Phillips, 1983). 

Another source concerning the selecting and purchasing of 
vendor-supplied training programs was presented by Martinetz 
(1986). Based on his experience at AT&T, he proposed a checklist 
form for program evaluation. No attempt was made to substantiate 
or relate the checklists to previously published research, theory, or 
models. Essentially, the reader is presented with two worksheets 
based on both a subject matter review and an instructional 
technology review. Section One, performed by a subject matter 
expert, is divided into three parts: technical content, relevance, and 
program design. Section Two, performed by a design specialist, also 
consists of three parts: program design, msriic. and administrative 
information. Summary sheets for each section are combined to rate 
the proposed program one of four ways: (a) acceptable as is, (b) 



acceptable with minor revisions, (c) acceptable with major 
revisions, or (d) unacceptable. 

A very limited explanation accompanies the two checklists. 
Unfortunately, the author failed to consider many of ihe questions 
and issues associated with the development, use, and validity of the 
materials presented. For example, is it valid for a subject matter 
expert to rate the program design if he or she is not trained in the 
program development process? How is the information from the 
cnacklists fed back into the program selection process? Are there 
trade-offs in using a simplified yes or no rating scale versus using a 
more complex one? A strength of Martinetz's checklists lies in the 
practical way in which the checklists were supposedly developed 
and used. Unfortunately, no data are provided on the actual 
effectiveness of the model. 

Several authors deal with the evaluation of programs within a 
specific industry or subject context. According to Honeycutt, 
Harris, and Castleberry (1987) the typical sales training program 
checklist includes such criteria as objectives, content, type, 
instructional methods end techniques, aids, trainees, trainers, 
administration, and evaluation. Objectives, content, methods and 
techniques, and eialuation were the four most important criteria 
they considered. 

Powers' (1983) model for designing sales training evaluation 
includes a step for conducting instructional analysis. This step 
requires listing the skills contained in the program and determining 
how they could be measured when translated to the actual 
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environment. Because his model requires that the measurement 
variables be quantifiable, the measures serve as a basis for 
experimental comparison, and statistical analysis can be used to 
fneasure the overall effectiveness of the ^i.ogram. In addition, the 
same data can be translated into monetaiy terms so that a cost 
benefit analysis can be performed. 

Focusing on the means by which organizations can improve the 
effectiveness of sending members to outside seminars, rather than 
how they can bring an outside program in-house, Kirkpatrick {1985b) 
proposed a five-step model. In his discussion of management and 
supervisory training and development programs, he stated that "the 
best way to judge the relative merits of a particular program is to 
evaluate them after they have been attended" (p. 40). Participants 
fill out a simple evaluation form, which is used to rate the program 
for other people who might be interested in attending it. 

The strength of Kirkpatrick's discussion lies in his 
consideration of the organizational factors involved in using outside 
programs. Although there are limitations to his remarks about 
selecting a program, other steps of his model, such as selecting 
participants, preprogram discussion, postprogram discussion, and 
keeping records, are useful. Need and desire are important to the 
participant selection process. Involvement of the student's 
supervisor in a preprogram discussion concerning the program and 
the individual's participation in the program is important. According 
to Kirkpatrick, the follow-up postprogram discussion should include 
a written evaluation, a written and/or oral summary of content, and 
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assigned projects that are based on the preprogram discussion. 
Records should be kept of who has attended what program. 

None of the models reviewed is complete. However, certain 
factors were important in more than one model. These factors 
include: (a) how well the subject matter or content of the program 
fits the identified training need, (b) the quality of the program's 
instructional design methodologies, (c) aspects of the purchasing 
organization, (d) aspects of the vendor's organization, and (e) ways 
to determine the effectiveness of the program, such as contacting 
previous buyers. Several of the models advocate certain processes 
by which the purchasing organization may better evaluate each of 
the factors that are important to the decision-making process. 
These factors serve as the basis for creating a model for the 
selection of vendor-supplied training programs in the next chapter. 
Sources of Externally Provided Programs 

How an individual or organization may find out about vendor- 
supplied training programs is also addressed in the literature. The 
process of choosing a program is compared to that of selecting a 
person to fill a job: the more choices there are, the more likely a 
good decision will be made (Kirkpatrick, 1985b). The following Is a 
list of potential sources for finding out about externally provided 
programs: 

• Direct mail brochures are targeted with surprising 
accuracy to professionals and prospective decision 
makers who might consider use of a vendor-supplied 
product. (Rogers & Voipe, 1984; Sredl & Rothwell, 1987) 
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College and university programs as well as vocational 
education institutions may provide other sources. 
Professional societies such as the American Society for 
Training and Development (ASTD) provide a number of 
sources: informal networking at meetings, the ASTD 
member information exchange that is cataloged 
according to professional expertise and interests, a 
computer network of subject information, and a 
publication in which many vendors advertise. 
Periodicals specific to an industry or topic may provide 
reviews of programs and advertising of specific 
programs. Vendors often use training industry 
publications, such as the Training and Development 
Journal (published by ASTD) or Training magazine 
(published by Lakewood Publications, Minneapolis, MN), 
as a place to advertise their products and services. 
Specialized publications often list available programs by 
topic. One example is the yearly Trainer's Resource 
(published by HRD Press, Amherst, MA). This two-volume 
publication lists programs by training topics and reviews 
available programs under each topic. Although the lists 
are not complete, this format provides more infcrrrtation 
about the program than do most other sources. Programs 
are reviewed according to intended audience, program 
description, delivery system, instructional strategies, 
recent users, cost, rental, preview availability and 
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vendor. Other examples are the yearly ASTD Buyer's 
Guide and Consultant Directory and Marketplacfl Directory 
(published by Training magazine). 

• Trade shows and conferenoes often include vendor 
exhibits and provide an opportunity for training buyers to 
meet professionals and consultants who have similar 
interests and experiences. 

• Reporting sftrvines such as that offered by ASTD's 
Trainet or the Seminar Clearinynouse International of St. 
Paul, Minnesota, will provide a listing of available 
programs to meet a particular topic. 

• Networking and asking people in other departments of 
one's own organization about sources of Information 
often yields Information about potential programs, 
vendors, and consultants. 

The above list of potential sources is a reasonably complete 
list of sources to find potential programs and vendors (Sredl & 
Rothwell, 1987; Cantwell et al., 1976; Rogers & Voipe, 1984). What 
these sources do not provide is a basis for evaluating the quality or 
effectiveness of the program in question. Although some sources 
are more complete than others, no source claims to provide a 
complete list of all, or even most, of the programs available for a 
given topic. Many of the sources would have the reader believe they 
offer a complete list, even if that is not true. Even though the above 
list Is a synthesis of (he sources mentioned in the literature, it too 
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is not complete. Further, the quality of a program cannot 
necessarily be inferred from the reference source. 

Instructional Development Systems 

Although a review of the literature on instructional 
development systems and program design components may provide a 
framework by which to assess programs according to acceptable 
instructional design methodologies, each of the components and 
processes used in program design and development may be 
considered from the perspective of program selection and 
evaluation. The process by which a model for program selection is 
developed would necessarily have some basis in the program 
development literature. 

A representative instructional design system that could serve 
as a bacis for cJeveloping a program selection model and instrument 
is the Training Technology System (TTS) proposed by Swanson 
(1987a). The five phases of the TTS are as follows: 

1 . Analyge. The analyze phase focuses on (a) separating 
training problems from nontraining problems and (b) 
defining precisely what people need to know and what 
they must be able to do to perform at work. 

2. Design. The design phase includes both program design 
and lesson design Program design is focused on 
matching the training program to organizational needs 
and constraints. Eight variables are used to focus on the 
lesson design process: (a) trainee readiness, (b) 
objectives, (c) content structure, (d) instructional 
sequence, (e) rate of delivery, (f) repetition and practice, 
(g) reinforcement and rewards, and (h) knowledge of 
results. 

3. Develop. In this phase lesson designs result in lesson 
plans for instructor-based training or story boards for 
media-based materials. Peer review and pilot testing of 
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training materials usually result in revision of the 
materials, lesson design, and/or program design. 

4. Implement . The implementation phase contains the 
program management plan and delivery of training. 

5. Control . The control phase includes (a) evaluating and 
reporting the effectiveness of training, (b) revising 
training, and (c) maintaining trainee behavior once 
trainees are back on the job. Evaluation, the major step 
in the control phase, consists of an effectiveness 
evaluation plan, tools for measuring training 
effectiveness, and the evaluation report. (Swanson, 
1987a) 

Worksheets for performing many of the tasks in the TTS are 
provided. 

This model is similar to previously developed models, for 
example, Tyler's (1949) that proposes (a) needs assessment, (b) 
setting of goals and objectives, (c) design of instruction, (d) 
management of the instruction, and (e) program evaluation. 

Meister (1986) suggested a word of caution concerning this 
discussion of models for the instructional design process. 
Differences exist between what the models present and what 
happens in actual practice. The theory-practice discrepancy in such 
models has been stated by Biookfield: 

Nowhere is this theory-practice disjunction more evident than 
in the realm of program development for adult learners. As a 
professor who has taught many program development courses 
to educators and trainers of adults, I can attest to the 
frequency with which participants in these courses (who are 
mostly practitioners with several years of experience) state 
that they "break the rules of good practice" or "disregard 
theory for the real world of practice (1986, p. 202). 

Regardless of likely differencos between theory and practice, 

the TTS is a suitable framework from which to develop criteria for 
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evaluating and selecting vendor-supplied programs. Criteria that 
are important in each phase of training development may be 
reviewed from the perspective of the decision maker who is 
responsible for selecting a program. 

Training Evaluation 
Although the focus has not been on the selection of pre- 
existing training programs, much has been written about training 
evaluation. Initially, the focus of training evaluation was on 
outcomes, more specifically, on quantifying the effectiveness of a 
program after its implementation. This has led to models and 
procedures concerned primarily with the effectiveness and 
implications of training outcomes. A more recent focus of the 
evaluation literature addresses issues that occur before the 
implementation of the program. Ideas from the research and theory 
of training evaluation literature provide content that is important to 
the questions addressed in creating a selection model and 
instrument. 

Before addressing the overall training evaluation literature, it 
is necessary to discuss the role that training evaluation can have in 
organizations. The purpose for which the evaluation is conducted is 
the primary factor in determining what models and methods the 
evaluator will use. The evaluation techniques, methods, and 
procedures will determine the evaluation outcomes. The focus of 
this investigation is guided by a primary emphasis on the selection 
process, and it is indirectly concerned with proving the cost-benefit 
ratio or effectiveness of the program to other members of the 
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organization. This investigation provides evaluation criteria for the 
decision maker to use during the part of the selection process that 
occurs before the implementation of a program, rather than focusing 
on after-the-fact evaluation. 

What Are the Purposas of Trainin g Evaluation? 

Why do organizations and individuals evaluate training? Based 
on a survey of 33 training organizations, Brandenburg (1982) 
determined thpt the seven top-rated functions of training evaluation 
were (a) to improve the training program, (b) to provide feedb?>ck to 
program planners and management, (c) to gain knowledge of 
employee skill levels, (d) to provide feedback to program 
participants, (e) to build status or prestige for the training unit, (f) 
to analyze the cost of activities; and (g) to study employee 
effectiveness. This suggests that the primary focus of training 
evaluation is to provide feedback for the program development 
process rather than to simply prove program effectiveness, to show 
cost-benefit data, or to determine a go/no-go decision. 

Evaluation techniques that provide feedback for program 
development and revision processes require a different focus than 
those associated with post-training evaluation. Rather than viewing 
evaluation as a relatively static task that occurs at isolated, pre- 
determined moments during the training program, evaluation can be 
seen ?is a dynamic, continuous process by which training programs 
and curriculum are updated and made more effective and efficient. 
Goldstein (1986) stated this as follows: 



34 

41 



For me evaluation is the systematic collection of descriptive 
and judgmental information necessary to make effective 
training decisions related to the selection, adoption, value, and 
modification of various instructional activities. The 
objectives of instructional programs reflect numerous goals 
ranging from trainee progress to organizational goals. From 
this perspective, evaluation is an information-gathering 
technique that cannot possibly result in decisions that 
categorize programs as good or bad. Rather, evaluation should 
capture ihe dynamij flavor of the training program. The 
necessary information will then be available to revise training 
programs to achieve multiple instructional objectives. 

It is possible to consider training evaluation as a succession 
of steps which provide information of better and better 
quality.... The point is that establishing the validity of training 
programs involves building a network that gives more and more 
Information with better and better controls so that the 
evaluator has more faith in the evaluation. 

Training programs should be considered dynamic entities that 
slowly accomplish their purpose in meeting predesigned 
objectives. Without systematic evaluation, there is no 
feedback to provide the information necessary to improve 
programs or qualify information to make decisions, (pp. viii. 
143. 175-176) 

Who Performs t^^ft Tf ainin? Fvalii^^t jQn? 

What people in the purchasing organization carry out the 
evaluation of potential training programs? This is important in 
order to know to whom a training program selection model and 
instrument should be directed. If the CEO of a company were to use 
the model, it might require a different design, or at least a different 
level of explanation, than if a person who is already trained in 
program design and development were to use it. 

In a survey of management training evaluation procedures in 
50 randomly selected Fortune-soo corporations, Clegg (1987) found 
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that the training staff was rssponsibie for evaluation efforts about 
80 percent of the time. An additional 8 percent of the companies 
used an ad hoc committee, which probably included representation 
from the training department. Outside consultants and specialists 
were used only 3 percent of the time. This indicates that there is 
almost no reluctance to having the group that is responsible for 
providing the training also provide the evaluation of the training. 

Brandenburg's 1982 study addressed the skills required for an 
effective training evaluator. Interpersonal skills ranked highest, 
followed by curriculum development, program development, and 
instructional-material development, respectively. This is 
consistent with the overall conclusions of the Clegg (1982) study 
that the function of training evaluation is to provide feedback to the 
training process, in terms of the perb^n providing the evaluation, 

smaller organizations are more likely to have an instructional 
designer or developer function as an evaluator, while larger 
organizations would more likely have a person with a broad 
range of statistical, measurement and development skills. 
This latter person, however, would not necessarily have sole 
responsibility devoted to evaluation tasks. (p.18) 

Sredl and Rothvell (1987) listed HRD practitioners, learners. 

and thf'rd-party evaluators as potential evaluators. Third-party 

evaluators could include managers, subordinates, or peers of the 

trainees, as well as consultants inside or outside the organization 

or HRD depa;1ment. In order to assure compatibility among programs 

supplied from an outside source, one company reported using an 

internal quality-control panel of people who work in the targeted 

training area. "The group screens new material and helps determine 
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whether it's compatible with what has been taught before" (Cothran, 
1987, p. 84). In addition to potential trainees, subject matter 
experts, design experts, and HRD people can serve on the selection 
committee. Various structures might be used to direct the group's 
activities. Rating instruments, checklists, or group interaction 
methods, such as the nominal group technique may be employed to 
stimulate discussion and to achieve a group consensus (Brinkerhoff, 
1988). 

In summary, the evaluator is usually a member of the HRD 
department. The evaluation may be performed by an individual or an 
ad hoc team. The choice of who will perform the evaluation depends 
on the organization ai d purpose of the evaluation. The involvement 
of representatives from the target audience appears important to 
the iQantitication of certain problems involving consistency among 
training programs from different sources. 

Models of Training FvalnatiQn 

The best known framework for training evaluation was 
proposed in 1967 by Donald L. Kirkpatrick (1975). His four-step 
hierarchy addresses the major elements in the evaluation process: 
Reaction- This lowest level is measured by participant 

surveys and addresses the question, "To what degree were the 

participants satisfied or unsatisfied with the program?" 

Learning. Test scores are used to measure the degree to 

which participants understood and learned the principles, 

facts, and techniques in the training program. 
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• Behavior. This third level determines the degree to 
which job behaviors actually changed as a result of the newly 
acquired skills, knowledge, or attitudes provided by the 
program. 

• Results . This fourth level addresses the tangible 
consequences of the training at the organizational level. This 
may be done by methods such a«5 an experimental assessment in 
which trained workers are compared with untrained workers or 
a cost-benefit analysis (Birnbrauer, 1987; Kirkpatrick, 1976, 
1983). 

By focusing on the organizational impact of training, Kirkpatrick's 
model provides a ba»is for moving beyond simple reaction-based 
evaluations. Indeed, Hamblin (1974) added a fifth level entitled 
ultimate valuft. This level goes beyond the experimental research 
that measures short-term results to include long-range issues such 
as how the training affects personal career goals and organization 
strategy (Hamblin, 1974; Sredl & Rothwell, 1987). 

Critics of the Kirkpatrick model focus on the outcome 
orientation it represents. Each of the four levels focuses on 
evaluating training that is already in place, rather than training that 
is being developed or selected. By addressing issues associated 
with the development of a training program, Kirkpatrick perpetuates 
the trial-and-error approach to program selection and 
implementation. If one purpose of evaluation is to provide the 
organization with input that can be used to revise, or even abort, a 
program before it incurs the higher costs of implementation or 
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piloting, the Kirkpatrick model provides no assistance (Brinkerhoff, 
1988). 

Brinkerhoff provided a more comprehensive model for 
evaluating HRD that begins with determining the training needs of 
the organization and includes assessing the long-term 
organizational impact and worth of the program. Based on the idea 
that the primary importance of HRD is training employees to 
ultimately benefit the organization, he developed the following six- 
stage model: 

1. Goal Setting (What is the need?) A need, problem, or 
opportunity worth addressing exists that could be 
influenced favorably by someone learning something. 

2. Program Dasign (What will work?) An HRD program 
capable of teaching the ne ded something is designed or 
located. 

3. Program Implementatinn (|s it working?) The 
organization successfully imploijents the designed 
program. 

4. Immediate Outcomes (Did they learn it?) The 
participants exit the program after successfully 
acquiring the intended skills, knowledge, or attitudes. 

5. Intermediate or Usa^e Outcomes (Are they keeping 
and/or using it?) The participants retain and use their 
new learning. 

6. Impact and Worth (Did it make a worthwhile 
difference?) The organization benefits when 
participants retain and use their learning. (Brinkerhoff, 
1988, p. 67) 

Brinkerhoff's model incorporates Kirkpatrick's outcome evaluation 
levels and Hamblin's organizational focus, as well as needs 
assessment issues. In addition, he addresses the development and 
instructional factors that are the focus of this paper. His model 
will be considered in greater detail. 
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An advantage of Brinkerhoff's model is that it identifies two 
critical decision points that are important to creating effective 
training solutions: (a) If the needs assessment is wrong, the entire 
program will be worthless, and (b) if a program with a critical 
design flaw is implemented or pilot-tested, valuable time and money 
will be wasted. By forcing HRD professionals to clarify the logic of 
a program from the conception and definition of a training need to 
the organizational payoff, the "six-stage model precludes defining a 
program as successful because it is popular, or because it is easy to 
teach, or because it uses state-of-the-art technology" (Brinkerhoff, 
1988. p. 68). 

By expanding the evaluation focus to include the developmental 
stages of HRD programs, the process by which to improve programs 
and results can be identified. Because it focuses primarily on the 
result, outcome information does not necessarily identify the source 
of the problem. This model: 

emphasizes a formative evaluation role and encourages the 
recycling of evaluation information from and to each of the six 
stages. In this way, all programs are made to wo.k as best 
they can, and good programs are made even better. 
(Brinkerhoff, 1988, p. 68) 

It emphasizes getting at the important information required for 

making good training decisions. Because no program is perfect, but 

rather represents degrees of acceptability across a number of 

factors, information is required about how and to what degree a 

program fits the decision maker's goals and the organization's needs. 
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stage II of Brinkerhoff's model addresses the program design 
aspects and Implementation issues of training evaluation that are 
Important to this study. This stage determines the point at which a 
design is strong enough to put it into operation. Similarly, this 
study addresses the issue of how to determine whether or not a pre- 
existing program is good enough to be implemented in the buyer's 
organization. This stage also may represent issues associated with 
assessing how a program could be customized to fit the unique 
characteristics in the buyer's organization. 

Key evaluation questions for Stage II include, what kind of HRD 
design might work best in a given situation? Is design A better than 
the alternatives? What is wrong with the design? Is the design of 
high enough quality to proceed with implementation? Procedures by 
which these questions can be answered include expert reviews, 
literature reviews, panels, checklists, site visits, pilot tests, 
participant reviews, observations, trainer and trainee feedback, and 
records analysis. 

The questions at Stage II have several common 
characteristics: they are future-oriented because training has not 
yet been implemented, and the answers are largely a result of expert 
judgment, not fact. Research on teaching and learning, program 
development, and training experience and expertise are irrportant 
for providing an informed judgment at this stage. Benefits of using 
a team approach in the Stage II evaluation include: 

1 ■ Commitmant and "buy-in " A systematic Stage II 

evaluation that solicits opinion and advice from those 
persons in the organization who will be most affected by 
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the HRD program engenders the "buy-in" phenomenon and 
increases the commitment to change that is required if 
change is to occur. 

2. Customer accountability and nnnnflrn A Stage II 
evaluation can be conducted to identify the expectations 
for and reactions to proposed program designs of such 
HRD "customers" as trainees and their immediate 
superiors. [This] demonstrates and ope rationalizes a 
concern for customer accountability and assures that 
customer concerns are considered before HRD takes 
place. 

3. Using and modeling participation Stage II evaluation 
affords an opportunity to model participatory methods. 
Carefully and systematically gathering trainee reactions 
to program designs and plans is one of the best and most 
direct methods of meeting this compatibility issue. Such 
efforts almost always yield ideas on how to make 
critical revisions to a design. Such involvement breeds 
political support and enhancos commitment to and 
"ownership" of the program among trainees. These 
efforts democratize HRD and tend to diffuse its control 
to increasingly lower levels in the organization. 

4. Facilitating tran'sfer of training . Stage II evaluation 
approaches can be used to systematically involve these 
key groups in the training design process, soliciting and 
using their input to revise designs. ..First, these groups 
will inevitably have good ideas on how to revise the 
training to make it more usable on the job. Second, the 
process of soliciting their input is a sort of covert 
training that creates knowledge and awareness crucial to 
transfer of training effects. 

5. Shaping expectations for sucoass . Stage II evaluation 
that involves key audiences for the program in a critical 
review of designs will be better able to inform those 
audiences about how the HRD program is supposed to 
work, the problems it will face, and what it hopes to 
accomplish. Such Stage II procedures, because they 
stimulate dialogue about HRD, will help articulate and 
shape expectations for HRD's performance. 

6. "Marketing" training in the organization. Human resource 
development leaders who make sincere and systematic 
Stage II evaluation efforts that involve, over time, key 
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audiences from all levels In the organization are 
marketing the HRD function in a legitimate and 
productive manner. These efforts make HRD visible, keep 
it responsive to organizational needs and interests, and 
educate const'mers as to its functions, problems, and 
benefits. (Brinkerhoff, 1987, pp. 73-77) 

These benefits of participative techniques in program development 

and delivery and mastery measurement are similar to those outlined 

by Garen and Daniel (1983). Yet the limitations due to the 

theoretical nature of this stage require movement into Stage III to 

determine the actual, practical outcomes of what has been 

developed. 

To document the program design, Brinkerhoff proposed using 
three worksheets In Stage II evaluation. The participant/outcomes 
analysis worksheet delineates who will receive the training, what 
are the immediate learning outcomes, what are the job-usage 
objectives, and what organizational benefits will occur if the 
training is successful. The components network worksheet shows 
the major process components of the program and how they work 
together to achieve the program's purpose. The input-process- 
output (IPO) worksheet gives the detailed operation of a component 
or subcomponent. On this worksheet, the inputs or resources that a 
particular component needs, the process by which the inputs will be 
converted into outcomes, and a listing of the outcomes that will be 
achieved are documented. 

Criteria critical to Stage II evaluation include the following: 

1- Claritv and Definition Stage II requires clear definition 
of (1) HRD needs, goals, and objectives at immediate 
learning, usage, and organizational benefit levels, (2) 
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HRD processes and methods, and (3) the resources and 
inputs planned to support the program. 

2. Theoretical Adequacy of the HRD DflRign Training 
designs must be theoretically sound. They should 
incorporate sound learning theory and reflect the current 
knowledge and recearch base about hov: people learn 
best. These criteria for theoretical soundness occur over 
a wide range of dimensions, including such aspects as 
the structure of a learning activity, the format (and even 
the color) of materials, the extent of interactions and 
feedback incorporated, and the adequacy of facilities. 
The fields of instructional psychology and design are 
especially pertinent to this criterion. 

3. Compatibility. A design must "fit" its environment and 
the culture of the organization of which it is a part. 
Considerations of compatibility would include such 
questions as: 

(a) Is the program consistent with other training 

programs and organizational priorities? 

(b) Is the program consistent with corporate culture, 
policy, and values? 

(c) Are program content and procedures compatible with 

trainee educational and social levels, values, and 
expectations? 

(d) Is the program consistent with the schedules, work 

demands, and personal practices of trainees? 

4. Practicality and Cost Effectiveness An HRD design must 
be economical and feasible. Evaluation of the cost 
criterion should be directed at each component in the 
program design. 

5. Responsiveness to Needs As a program design is nearing 
its final shape and specifications, it is wise counsel to 
"revisit" the initial goals and ask whether this HRD 
program is likely, in fact, to meet the goals initially set. 

6. Superior to Alternatives. Training resources are limited, 
meaning that the problem is rarely that of getting the 
best HRD that money can buy; rather, it is usually that of 
getting the best HRD that the least money can buy. To 
put it briefly, alternatives must be considered and 
systematically compared during Stage II. HRD 
professionals should be prepared to argue not only that 
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one particular design is good but that it is better than 
contending alternatives. 

7. Adult-Learning Practices. Program designs should 
reflect state-of-the-art practice when possible and 
should cenainly avoid obsolete or outmoded methods. 

8. Leoalltv and Ethins Criteria regarding ethics and 
legality are absolute and must not be compromised. 
(Brinkerhoff, 1987, pp. 85-89) 

In summary, although Brinkerhoff does not consider the 
selection of vendor-supplied training programs directly, his Stage II 
Program Design considerations are relevant to this discussion. The 
benefits of performing Stage II design are similar to the potential 
benefits of a vendor-selection model. Worksheets are a powerful 
method for synthesizing data, and they provide the basis for an 
effective selection instrument. Criteria that are important at this 
stage may be similar to criteria that are important to consider in 
selecting a vendor-supplied training program. 

Fcrmative/Summati ve Evaluatinn 

Front-end, formative, and summative evaluation processes 
represent distinctive elements in the evaluation literature. The 
distinctions between the three are based primarily on the time the 
evaluation is to take place and for whom the evaluation is intended. 
Evaluation of the needs assessment process occurs before the design 
is started, and it represents front-end evaluation. The first 
outcome of the needs assessment is the basis for determining that 
the perceived need can be met effectively with a training solution 
(Harless, 1975). The second outcome of the needs assessment, 
provided training is determined to be the most effective strategy to 
meet the need, is a detailed understanding of the training objectives 
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that must be achieved in order for the program to be effective (Sredl 
& Rothwell, 1987). 

Formative evaluation provides feedback to the developer in the 
initial stages of program development. As conceived by Scriven 
(1967), formative evaluation is used to determine if the program 
will adequately meet the training needs for which it was designed. 
The focus is on testing a program's effectiveness before it is 
implemented. Formative evaluation is not limited to design 
evaluation, such as expert reviews for conformity to instructional 
design theory and accuracy of subject matter. Such empirical 
research methods as pilot-testing or performing behavioral 
rehearsals as a pretest of the materials units of the program may be 
used (Dick & Casey, 1978, 1985). Time and expense are key factors in 
determining the methods ultimately chosen. An expert review may 
be accomplished in days by one person, whereas a pilot test may 
require several weeks of planning and the involvement of a team 
(Foshay, 1984). 

Summative evaluation focuses on the efff^ctiveness of the 
program after its implementation (Anderson, Ball, Murphy, & 
Associates. 1975). Rather than focusing on data for the developers 
of the program, summative evaluation addresses issues that are 
more important to policy makers (Foshay, 1984); for example: How 
well dia the participants learn the program objectives? Was the 
learning applied on the job? 

Satisfactory reviews of the program during the formative 
evaluation process does not ensure that it will meet summative 
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evaluation standards. Thus, both formative and summatlve 
evaluation methods are required. However, formative evaluation 
should be performed before summative evaluation is begun 
(Goldstein, 1986). 

Foshay cast the distinction between formative and summative 
evaluation in terms of quality control and quality assurance based on 
the differing needs of the public and private environment. 

The business context changes the nature of the evaluation 
design so that many aspects of the conventional 
formative/summative distinction no longer apply. Structuring 
the training enterprise as a business places special 
requirements on the evaluation which differ from those of the 
public sector. Significant factors are the structure provided 
by the vendor-client contract, the importance of cost- 
effectiveness considerations, and the usefulness of evaluation 
as part of the design and project management systems. When 
taken together, these factors lead to an evaluation system 
which is quite different from those commonly described for 
public-sector projects, even though many of the underlying 
principles are the same. (1984, pp. 15, 17) 

Foshay equated summative evaluation with quality assurance and 

formative evaluation with quality control. To the training vendor, 

redefining summative evaluation as quality assurance moves the 

focus from the final product provided to providing a means by which 

the vendor can assure the client that defined quality standards have 

been met throughout the development process. The quality of the 

process the vendor used to develop the training program is as 

important as the product. Unlike summative evaluation, quality 

assurance is determined by the vendor before the program is 

implemented in the client organization. Essentially, quality control 
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is the same as formative c;valuation. Business concerns SuJh as 
time and money increase the likelihood of using cost-effective 
formative evaluation approaches, sjch as expert reviews, instead of 
empirical trial-and-error methods, such as pilot trials. 

The critical elements of this discussion that can be used here 
include the breakdown of the formative/sum mative distinctions in 
the business setting and the consequent introduction of adr^itional 
factors into the evaluation process. These factors include (a) an 
increased emphasis on cost effectiveness, ^b) the importance of 
creating evaluation data that are timely and useful to the decision 
makers, and (c) an evaluation process that is compatible with the 
company's management structure. These are factors that result 
from the business nature of the client/vendor business relationship 
(Foshay, 1984). 

Instructional Design and Delivery 
In the literature on instructional design and delivery, a number 
of evaluation models have been proposed based on the need to make 
instructional design decisions rather than management decisions. 
One author stated that there is no lack of such methodologies 
(Morrison, 1986), but another stated that the most serious problem 
in training evaluation has been "the failure to even consider the 
instructional methods" ano various components of the training 
program (Goidstein, 1986, p. 113). In this section key aspects of the 
literature concerning the development stages of training programs 
are reviewed. The framework by which the instructional design 
literature will be referenced is based, in part, on the key areas 
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suggested by the previously reviewed literature on the selection of 
vendor-supplied training programs and on the phases of the Training 
Technology System. Instructional design systems, such as the TTS 
and other evaluation models, use various design evaluation 
procedures and assume the existence of design-evaluation 
literature. Comparing program content to the specified training 
needs Is considered first. This dimension corresponds to the 
question of how well the program addresses the identified training 
needs of the purchasing organization. 

Ic>sues associated with program design and development will 
be considered in two areas: instructional system factors and 
implementation factors. Instructional system factors include 
objectives, trainee readiness and trainee characteristics, structure 
of content, instructional sequence, application ot adult learning 
principles, and anticipation of transfer-of-learning problems, 
implementation factors include rate of delivery, repetition and 
practice, knowledge of results, reinforcement and rewards, delivery 
methods, media plantiing, and written materials (based on Smith, 
1983a). These dimensions correspond to the design and development 
phases of the TTS. Literature pertaining to program, lesson design, 
and training materials development are referenced in the Training 
TechnolQQv System Rihlingrgphy (Swanson, 1987) and in the ASTD 
Reference Guide to Professional Training Rnlas and Compfltftnniflg 
Vol. II by Sredl and Rothwell (1987). 

Contant 
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■^he content of the program is the key focus of the initial 
design effort. What knowledge, skills, or attitudes will be 
addressed in the training? The critical strength of the TTS system 
considered above is its in-depth analysis phase. The precision with 
which the analysis of work behavior is performed, the determination 
of organizational as well as individual needs, and the focus on 
organizational causes and motivational elements of performance 
provide the solid foundation that is required for successful 
implementation (Swanson, 1986). The outcomes of the analysis 
phase provide the criteria by which to assess the effectiveness of 
the training in the latter phases (see Swanson & Gradous, 1986). 

When the decision maker considers program content, some 
issues become apparent: To what degree does the program contain 
the content specified in the analysis phase? Does the program 
contain material not required by the analysis? The relationship 
between the stated requirements of the program and the program 
itself forms a two-axis matrix with four possible values: (a) 
content that is required according to the analysis and that is 
addressed in the training program represents a measure of content 
relevance, (b) content that is not required according to the analysis 
and that is not addressed in the training program is also considered 
to represent a measure of conter>'r relevance, (c) objectives from the 
needs assessment that are not covered by the training program 
represent content deficiencies, and (d) Information covered in the 
training program that is not required according to the analysis 
performed represents content contamination. The following diagram 
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shows the possible relationships among program content and 
assessed objectives (see Goldstein, 1986). 
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Figure 1 --Relationships Between Content and Assessed Needs 

To the degree that the content of the program represents the 
knowledge, skills, and attitudes required, the program may be 
considered content-reievant. This measure provides a basis for 
utilizing training in the most time-effective manner. Not provided 
by an evaluation of content relevance is information about how 
successful the program was in training the participants and how 
well the participants were able to transfer the training to their jobs 
(Goldstein, 1986; Guion, 1977). 

Because content relevance assumes that the information 
presented in the program is accurate and current, procedures for 
determining content validity may require a review by subject matter 
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experts. Several of the evaluation models referenced above discuss 
such reviews ( Cantwell, et al., 1976; Foshay, 1984; Martinetz. 1986; 
Rogers & Voipe, 1984). 

Instructional System Factors 

Two sections of the design evaluation literature that were 
reviewed include the methods and the techniques available to the 
designer for presenting the program content. Issues such as trainee 
readiness, content structuring, instructional sequencing, rate of 
delivery, repetition and practice, testing, measurement of results, 
and reinforcing and rewarding the training outcomes are key design 
variables identified in the Training Technology System (Swanson, 
1987a). Other design variables mentioned in the literature include 
delivery methods, implementing adult learning theory, and 
facilitating transfer-of-learning. 

Critical to any discussion of design variables is the 
assumption that each of the variables mentioned contributes to the 
overall effectiveness of the program. However, research has not 
been able to verify what methods will lead to effective instruction 
(Tobias, 1987). Various suggestions have been made about why a 
connection between design and delivery variables and training 
outcomes has not yet been verified. According to the trait- 
treatment explanation, the lack of significant findings is the result 
of a failure to consider how the traits of individual learners affect 
the treatment. Differences in learning styles may alter the 
effectiveness of different instructional methods. These and other 
possible explanations have been reviewed by Smith (1983a, 1983b). 
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In response to these findings, Smith conducted a 
comprehensive review of the literature to determine what 
instructional variabi9S are common to instructional delivery 
systems (Smith & Currey, 1983). These variables, used in the 
Training Technology System, are discussed in detail in the following 
sections. 
Qbiectivfls 

Objectives have been the focus of considerable inquiry and 
research. They may be defined by type (cognitive, affective, and 
psychomotor), by scope (individual, job, organization, or society), 
and by time (end of unit, end of program, life) (Sredl & Rothwell, 
1987). In the Training Technology System, objectives are 
categorized on the basis of type or learning domain. The three 
learning domain categories of objectives include behavioral 
(psychomotor and observable skill), affective (attitudinal), or 
cogniiive (knowledge) objectives. Given that objectives may overlap 
regarding the learning domain, the dominant domain is used to 
categorize objectives. Choosing the correct domain depends on 
determining the learning domain that differentiates between expert 
and less than expert performance for that objective (Swanson & 
Gradous, 1986). 

Objectives also may be classified in terms of specificity or 
vagueness. At the program and subprogram level, objectives should 
be specific (Sredl & Rothwell, 1987). To be specific, an objective 
must include the following three elements; 
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1. The objective should specify a performance component 
(l^nowledge, skill, or attitude) identifying what a 
participant must do. 

2. The conditions under which the result of the objective 
will be achieved should be given. 

3. The criteria by which the outcome of the objective will 
be measured as successful should be given (Davies, 1981; 
Mager, 1962). 

Four common pitfalls in writing objectives were identified hy 
Davies (1981): 

1. Objectives are stated in terms of what the instructor is 
going to do, not the student. 

2. Objectives are based on the teaching strategy and not on 
the behavior in which the student is being trained. 

3. Objectives use high-sounding impressive words that 
don't mean much. It is better to use straightforward, 
simple English in writing objectives. 

4. Objectives fail to identify performance in clear enough 
terms. The action verb is ambiguous as opposed to being 
specific (i.e., to understand versus to name). The content 
is not well defined, (pp. 138-139) 

Pitfalls 1, 2, and 4 could be objectively assessed by a reviewer, and 
thus they could be included in a systematic evaluation process. 
Assessing the writing style of an objective appears to represent a 
more subjective evaluation process in which (here could be 
considerable ambiguity concerning borderline oases. 
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Trainee ReadinftRg anH Trainfte Charactarifitinft 

Most programs have been developed for a target populntion. 
The better the fit between the targeted audience for which a 
program has been developed and the actual participants, the more 
likely it is that the training will be effective. This relationship 
includes the concepts of trainee readiness and trainee 
characteristics. 

Trainee readiness pertains to "the stimulus content with 
which the learner is already familiar" (Smith, 1983a, p. 21). A 
pretest can be givian to prospective trainees to determine the 
knowledge and expertise they bring to the training. Perhaps the 
content can be initiated at a higher abstraction level that is 
familiar to the learners (Ausubel, 1963; Smith, 1983a). 

Closely related to trainee readiness is the concept of trainee 
characteristics. Whereas trainee readiness refers to knowledge and 
work experience shared by the targeted audience, trainee 
characteristics include other factors associated with training 
delivery. The Training Technology System asks the following 
questions concerning the characteristics of the targeted audience: 
What is the total number of trainees? How many trainees will be 
trained at one time? What is their education level? What type and 
amount of work experience do they have? (Swanson, 1987b) 

In order for the decision maker to select a vendor-supplied 
program, information is required about both the target audience fo' 
which the program was designed and the audience that will receive 
the training. The trainee characteristics of the target audience for 
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which a program was designed can be determined in several ways. 
The need the program is intended to meet may define the audience, 
such as an orientation program for new employees. Prerequisite 
knowledge and skill may be stated in the descriptive literature 
presented to potential buyers or in the introductory sections of the 
trainee materials and leader guides (Sredl & Rothwell, 1987). 
Trainee characteristics may be included in the assessment work 
performed by the vendor preceding the program development process. 
Structuring the Content 

Content structure describes how the designer presents the 
material so that the student can integrate the information with 
what was previously known. "Would the presentation be improved by 
presenting the material around a logical or a conceptual model?" is 
the key question (Swanson, 1984; 1987b). One form of structuring 
content involves the principle of scaffolding, in which each of the 
content elements or subsystems of the training is subsumed into a 
unified concept or overall system (Ausubel, 1963; Shoemacker, 1969; 
Smith, 1983a). For example, methods of troubleshooting personal 
computers may be graphically structured according to a picture of 
the parts within the system. What is important is that the structure 
employed allows the student to remember (encode) and later access 
(decode) the information after the training is completed (Smith, 
1983b). 

Other examples of structuring content involve using either an 
analogy or an advance organizer. An analogy can provide the means 
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for the learner to apply the new knowledge by relating It to already- 
acqulied schema. An advance organizer is: 

an instructional method which is positioned prior to the body 
of the instructional material and presents an overview of the 
instruction at a higher level of abstraction. The purpose of the 
organizer is to provide the learner with an organizational 
framework or schema on which the learner can build the 
details of the lesson. For example, in a course designed to 
provide content experts the skills and knowledge necessary to 
design and develop effective training programs, an 
introductory overview of the generic instructional design 
model is provided. An effective advance organizer must 
provide an organizational structure which accurately reflects 
or subsumes the details of the lesson for a learner who 
otherwise would not have the orienting structure. (Clark & 
Clark, 1984, p. 3) 

Unfortunately, the limited research concerning the 
effectiveness of structuring has been equivocal. Research on 
advance organizers has yielded inconsistent findings. This 
circumstance may be due to the failure of the researchers to 
consider differences in learners and the goals of the learning task 
(Mayer, 1979). The importance of the advance organizer could be 
dependent on variables not yet controlled for by the researchers. 
These variables could concern how well the advance organizer 
captures the instruction that it is intended to synthesize and the 
level of knowledge the trainee has prior to beginning the training 
(Clark & Clark, 1984). 
Instructional Sequencing 

Instructional sequencing addresses questions about where to 
begin the instruction and how to teach the material at the 
presentation stage. How can the activities and instruction methods 
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be organized best to communicate the content to the student? 
Briggs (1967) emphasized the importance of sequencing-arranging 
the content into hierarchies--so that new l^nowledge or sl^ills can be 
systematically integrated with previous knowledge and skills (also 
see Smith, 1983a). 

Various methods of sequencing have been proposed: 

1. Logic provides a commcnly used sequencing principle. 
Ordering the program based on the chronology of the 
elements from past to present is an example of this. 
Following a learning hierarchy in which skills are 
structured from simple to complex is another type of 
logical sequencing. 

2. Procedural steps in the order required to perform a 
task is another basis by which to sequence instruction, as 
in following a recipe. 

3. Whole-to-part learning may be accomplished by first 
presenting a model and then considering each aspect of it. 

4. Key questions around which to plan the instruction 
may provide a challenge for the learner and stimulate his or 
her curiosity. (Sredl & Rothwell, 1987) 

5. Inductive learning may be used when experiential 
learning is important. Through this approach, the 
participants discover learning for themselves by following 
a five-stage cycle of (a) experiencing or performing an 
activitv, (b) sharing reactions and observations, (c) 
processing and discussing the patterns and dynamics of the 
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experience, (d) generalizing and inferring principles about 
the event, and (e) applying and planning more effective 
behavior based on the new knowledne. (Pfeiffer & Jones, 
1973-1979) 

In the end, there is no one right method by which to sequence 
instruction. 

What is important is that approach match intended purpose. If 
the idea is to build skills, procedural structures or learning 
hierarchies are probably most appropriate. If the idea is to 
help learners adapt to life problems, then a problem-oriented 
approach is probably be^t. (Sredl & Rothwell, 1987, p. 116) 

Principles of Adult I flaming 

Many of the principles of adult education can be adapted to 
Increase a program's overall effectiveness. A group of faculty 
members and experienced practitioners at Columbia University 
reviewed the literature and reflected on their experience in teaching 
sdults. Their work resulted in eight principles for effective adult 
training that serve as a summary for much of the research 
concerning adult education (Brookfield, 1986; Knowles, 1980; 
Mezirow, 1981). Those principles are: 

1. The presence of a climate of respect for adult learners 
(physical, social) 

2. Reliance upon a collaborative mode of learning (including 
design, implementation, and evaluation) 

3. The fostering of progressive fearner self-directedness 
and empowerment 

4. Drawing upon learner experiences as a point of departure 
for learning experiences 

5. Learning realized through a participative environment 
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6. An emphasis on critically reflective thinking and 
awareness of social and cultural norms (contextual 
factors) 

7. A concern with learning for zWon (including decision- 
making, behavioral change, learning to learn, collective 
action) 

8. The fostering of problem-posing and problem-solving 
based on real problems in the lives of the learners 
(McAllaster, 1987, p. 25) 

Other sources on the subject of adult learning principles include 

Sredl and Rothwell (1987), Laird (1985), and Zemke and Zemke 

(1981). 

Transfer of Learning 

Perhaps the most important measure in the evaluation of 
training programs concerns the transfer of learning to the 
participant's job. If the knowledge, skills, or attitudes developed in 
the program cannot be utilized to improve job performance, the 
training has no value. The training may even have a negative effect 
if the on-the-job application of the new learning frustrates, 
demoralizes, or demotivates the participants (Sredl & Rothwell, 
1987). During the planning stage of program development, designers 
question internal and external conditions of the organization that 
could prevent the trainees from applying what they have learned to 
their jobs. Is the program content consistent with the culture of the 
organization, the work group norms, the role requirements, and the 
individual beliefs and values? At the program design and delivery 
stages, the attention of learners should be focused on identifying 
forces that may assist or prevent them from applying what they 
have learned (Sredl & Rothwell, 1987). 

Implementation Factors 
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The implementation factors of training considered here include 
rate of delivery, repetition and practice, knowledge of results, 
reinforcement and rewards, delivery methods, media planning, and 
written materials. 
Rate of nfllivfliv 

It is important to consider what the rate of learning should be 
and how big each instructional "piece" should be. Obviously, if the 
instruction proceeds too fast, participants will become frustrated, 
which will lessen the overall effectiveness of the training. If the 
instruction proceeds too slowly, participants may fail to pay 
attention, which, likewise, will lessen the program's overall 
tjffectiveness (Smith, 1983a; Swanson, 1987b). 
Repetition and Prantira 

Repetition and practice are also critical factors for training 
effectiveness and for increasing the transfer of the training to the 
job. Generally, the degree to which information is used and 
practiced in training determines how well the information will be 
used and retained on the job. Focus should be given to what the 
practice will consist of, how much practice is required, and how the 
practice will be remediated (Smith, 1983a; Swanson, 1987b). 
Testing and Measurinty Performanna 

Testing and measuring the participants on how well they have 
accomplished training objectives is conducted for at least four 
reasons: (a) to assess entry-level problems or difficulties of 
students, (b) to motivate the participants, (c) to identify problems 
with the delivery or design of the training, and (d) to provide the 
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participant and InFtructor with feedback on how we'l performance 
compares to the original instructional objectives. Certain types of 
tests or measurements work better with certain types of objectives. 
In the cognitive domain, test items may include true-false, multiple 
choice, ossay, fill-in-the-blank, matching, and oral response. 
Affective objectives can be assessed with essay, oral response, and 
projection types of items. Often, it is best if psychomotor 
objectives are demonstrated (Denova, 1979; Sredl & Rothwell, 1987). 

The critical questions of, "What will be used to assess trainee 
performance?" and "How will this information reach the trainees?" 
are critical to assessing a program according to its use of 
measurements and tests to provide feedback (Smith, 1983a; 
Swanson. 1987b). In reviewing training programs, thj decision 
maker should take care to ensure that test questions or 
measurer. )ent methods are consistent with the initial objectives and 
assessed needs (Rogers & Voipe, 1984). The high satisfaction 
ratings of participants should not be mistaken as a measure of 
participants' actual achievements (Cantwell et al., 1976). 
Motivation. Reinforcement, and Rawards 

R'^inforcement and rewards for training outcomes are powerful 
techniques for motivating students both during and after the 
training. When the rewards are changed from explicit to implicit, 
the likelihood that new behaviors will be implemented and persist 
after the training increases. Important questions to ask are: What 
reinforcement or rewards are offered in the program? How wil' 
they be applied (Smith, 1983a; Swanson, 1987b)? 
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Motivation is probably as important a force as ability when 
determining on-the-job performanco. A willing learner who has 
average ability may be much easier to train than an unwilling 
learner who has superior ability. At least six major factors have 
been idertifled and supported by research as having a significant 
impact on learner motivation (Wlodkowski, 1985a, 1985b). If each 
factor is given major considcrano.i at the appropriate point in the 
program, it can be used to maximum effectiveness. This research is 
summarized as follows: 

Beginning: 1. Attitude. The learner's attitudes toward the 

general learning environment, instructor, 

sjbject matter, and self. 
2. Need. The basic needs within ihe learner at 

the time of learning. 
Middle: 3. Stimulation. The stimulation processes 

a ecting the learner via the learning 

experience. 
4. Affect. The affective or emotional 

experience of the learner while (earning. 
Ending: 5. Competence. The competence value for the 

learner that is a result of the learning 

behavior. 

6. Reinforcement. The reinforcement value 
attached to the learning experience for the 
learner. (Wlodkowski, 1985b, p. 4) 

Each motivation facto' represents an aspect of the program that the 
buyer may consider when evaluating a program. For example, how 
does the program continuously ;»iimulate the learners? 
IfLStructor-Led Group Pra sentatjon Methnris 

Delivery methods will be discussed in two sections: 
instructor-led group presentation formats and media-based delivery 
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methods. Methods of delivery are used in instruction for several 
purposes: 

1. Methods represent the means by which the content of the 
program is delivered. 

2. The method may evoke insight and may motivate and 
maintain the interest of the participants. 

3. Certain methods can help students to relate the training 
to their jobs. 

4. Methods may provide a safe environment in which to 
practice a newly acquired ski'l or behavior. (Sredl & 
Rothwell, 1987) 

A comprehensive listing of instructor-led group presentation 
formats is presented in this discussion based on Laird (1985), 
Davies (1981), Wenig (1978), Ande-^on (1983), Goad (1984), and 
Sredl and Rothwell (1987). 



1. 


Lecture and Lecturettes 


2. 


Unstructured Reading Assignments 


3. 


Structured Reading Assignments 


4. 


Demonstrations 


5. 


Field Trips or Excursions 


6. 


Note Taking 


7. 


Open-Forum Discussions 


8. 


Question and Answer Sessions 


9. 


Performance Tryouts 


10. 


Brainstorming 


11. 


AcTion Mazes 


12. 


Case Studies 


13. 


Jigsaws 


14. 


In-Baskets 


15. 


Incident F'rocess 


16. 


Team Tasks/Buzz Groups 


17. 


Agenda-Setting Buzz Groups 
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18. 


Fishbowls 


19. 


Role Plays 


20. 


Reverse Role Plays 


21. 


Doubling Role Plays 


22. 


Programmed Instruction 


23. 


Structured Discussions 


24. 


Panel Discussions 


25. 


Rotation Role Plays 


26. 


Simulations 


27. 


Games 


28. 


Clinics 


29. 


Critical Incidents 


30. 


T-Groups 


31. 


Hot Role Plays 


32. 


Skits 


33. 


Cognitive Networks 


34. 


Behavior Modeling 


35. 


Conferences 


36. 


Workshops 


37. 


SvnriDOsia 

\^ Till w w 1 


38. 


Organization Development Gatherings 


39. 


Huddle Groups 


40. 


Delphi Procedures 


41. 


Nominal Group Techniques 


Media Planning 



Training media represent vehicles for delivering instruction to 
ihe learners. The questions about media consider whether certain 
media are more effective for presenting certain types of 
information with certain types of instruction, what effect the media 
has on instructional outcomes, and what constitutes a good 
application of media in a program. This section includes a 
descriptio. of the research on the use of media in training programs. 
The goals are to provide an understanding of various media available 
for training and to highlight information that is pertinent to proper 
use of the media (Sredl & Rothwell, I987^. 



65 



1^ 



72 



The research literature i.idicates that "media do not under any 
circumstances influence performance. There is clear evidence that 
in other than the most obvious cases, any medium will handle any 
subject matter content effectively" (Claris & Claris, 1984, pp. 1-2; 
Jamison, Suppes, & Welles, 1374). Some of the problems in 
researching the connection between media and performance also 
represent potential biases that must be considered when selecting 
programs. When newer media are compared to more traditional 
media, research results often are biased by the use of more 
effective instructional strategies in programs that include newer 
media. When instructional strategies are held constant, it has been 
proven that there is no difference between electronic and live- 
trainer presentations. Another research error is the failure to 
control for the novelty factor associated with newer media. In time 
periods of less than four hours, a slight increase in learning can be 
shown with novel media; however, this difference disappears after 
approximately four hours. "The use of a technologically complex 
delivery system does not, of itself, insure the quality of the 
instruction" (Clark & Clark, 1984, p. 2). In making purchasing 
decisions, the decision maker should take care to ensure that any 
advantages assumed about the more novel media are justified 
regarding additional expense and special requirements. 

Several of the potential benefits of using media properly are 
as follows: 

1. Costs associated with mediated programs may be 
considerably less that those required for classroom 



delivery. Instructor-dependent methods require ongoing 
delivery costs. Mediated programs costs are focused 
upfront during the development stages. 
Flexibility in timin9 and rieliverv may be an important 
benefit. Self-instructional formats may allow the 
employee to perform the training during off-work hours 
or it may help the trainer to meet timing requirements in 
a situation in which training is offered simultaneously 
at multiple locations. 

Certain media formats may provide more reliable and 
consistent delivery than other formats. For example, 
computer aided instruction will follovv a consistent, 
predetermined format whereas instructor led training 
will vary from instructor to instructor and even delivery 
to delivery by the same instructor. 
Media may improve learning by providing a variety of 
stimuli to the instructional process. Learning research 
has shown that 80 percent of learning occurs through 
sight, 10 percent through hearing, 5 percent through 
touch, and 5 percent through smell and taste. The more 
senses that can be used to deliver and reinforce the 
training, the more likely the learning will be retained 
(Clark & Clark, 1984; Sredl & Rothwell, 1987). 
The use of media in instruction may serve to place the 
instructor in the role of a leaminij managflr or facilitator 
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rather than being the lecturer or a similarly less 
interactive delivery role (Clark & Clark, 1984). 

6. Visual media provide a controlled model of the training 
behaviors being studied, which may result in vicarious 
learning and learning through modeling and imitation 
(Clark & Ci .rk, 1984). 

7. Trainees can replay the instruction as often as required 
to learn detailed information or review modeled 
information until they can successfully perform the 
behavior (Clark & Clark, 1984; Sredl & Rothwell, 1987). 

An important lesson, based on reviewing the literature and 
research concerning the use of media in training, is that the program 
should determine the media, not the other way around. The learning 
method that is the best method based on program objectives should 
be the basis for media-selection decisions. If a mistake is made, 
and the media dictate the design specification, then "the media tail 
wags the training method dog" (Clark & Clark, 1984, p. 3). Thi<> 
lesson is as important for the decision maker who selects a vendor 
program as it is for the program designer. Gilbert (1960) stated it 
this way: 

If you don't have a gadget called a teaching machine, don't get 
one. Don't buy one; don't borrow one; don't steal one. If you 
have such a gadget, get rid of it. Don't give it away, for 
someone else mighi use it. This is a most practical rule, based 
on empirical facts from considerable observation. If you begin 
with a device of any kind, you will try to develop the teaching 
program to fit that device, (cited in Goldstein, 1986, p. 20) 
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It is important that the medium matches both the learning 
method and the ultimate purposes of the program (Goldstein, 1986). 
"The best available basis for the needed matching of media with 
objectives... is a rationale by which the kind of learning involved in 
each educational objective is stated in terms of the learning 
conditions required" (Briggs, Campeau, Gagne, & May, 1967, p. 3). 

In summary, it is clear that administrative and logistical 
considerations, such as budget constraints, location of learners, and 
accessible equipment, do influence the selection of media. The 
instructional outcomes are much more the result of design 
strategies and instructional methods, than of media selection. 
Although research may not support differences in training outcomes 
due to the effect of media, the importance of media to the training 
program should not be underestimated (Sredl & Rothwell, 1987). 

Many of the questions associated with the selection of 
appropriate media during the design and development phases of a 
program are important considerations in the program-selection 
process. For example, should an overhead projector be used or 
should the instructor take the extra time to write major points on 
the blackboard? Sredl and Rothwell (1987, pp. 128-141) summarized 
these questions and other considerations associated with various 
media in the following list: 

a Overhead Transparencies 

b. Opaque and Rear-Screen Projectors 

c. Chalkboards, and Ceramic, Felt and Magnetic Boards 

d. Flipcharts 

e. Slides 

f. Filmstrips 
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g. Motion-Picture Filn 3 

h. Videotapes 

I. Interactive Videodisks 

j. Audiotapes 

k. Printed Material 

t. Models, Simulators, and Real Equipment 

m. Computer Software 

Instructional Materials 

Printed instructional materials represent the most commonly 
used media for instruction. Because of the importance of written 
materials in vendor-supplied training programs, they will be 
considered here in greater depth than other media. Sight is the 
major sense by which people assimilate information, thus, printed 
materials represent a critical medium for enhancing learning. 
Properly designed materials can add a professional flair to a 
program, can reinforce information presented by other means, and 
can provide participants with a way to review important concepts 
after the program is completed. Poorly designed and poorly prepared 
printed materials may detract from the learning presented in the 
training program. 

In Models for Excellence. McLagan (1983) mentioned the 
following examples of printed materials that may be used in 
training: 

1. Exercises, Workbooks, Worksheets 

2. Teaching Guides 

3. Manuals and Job Aids 

4. Tests and Evaluation Forms 

5. Written Role Plays, Simulations, Games 

6. Written Case Studies. 



Of concern to the present study are two issues: "What are the 
critical factors in evaluating printed materials?" and "What printed 
materials are important in a complete and effectively designed 
vendor-supplied training package?" The framework for this section 
will be considered on the basis of five factors identified by Burbank 
and Rett (1986) as important to using printed materials effectively: 

1. Content 

2. Writing Style 

3. Typography 

4. Illustrations 

5. Page Organization 

Content of printed materials includes aspects similar to the 
previous discussion of design factors: Do not include extraneous 
material, include all required material, and connect new ideas to 
previous knowledge. Issues that are specific to printed matter 
include: 

1. Employing simple cueing devices that assist and direct 
readers as they move through the material. 

2. Include an overview at the beginning of each section, as 
well as a summary at the end to identify important 
points. 

3. Use questions to focus learner attention on key ideas. 

4. Include examples and non-examples to clarify concepts 
better. (Burbank & Rett, 1986, p. 5) 

Writing style can be evaluated using the following questions 
as guidelines: 

1. Is the information presented in short, concise 
statements? 

2. Is the reading level appropriate to the learners? 
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3. Is the vocabulary concrete and familiar to the learner? 

4. Are new terms defined and examples given when 
possible? 

5. Is wording consistent throughout the document? 

6. Is active language used (particularly active present 
tense verbs)? 

7. Are long lists divided into groups to facilitate 
remembering? 

8. When the user must make choices, is continuous prose 
avoided and diagrams used instead? 

9. Are visual or verbal analogies used to clarify key points? 
(Burbank & Pett, 1986, p. 6) 

Typography deals with how legible the printed materials are 

and whether they conform to the findings of typographic research. 

Important aspects of thi** factor include: 

1 . Type Size. Sizes smaller than 9 points are hard to read, 
while sizes larger than 12 points tend to be viewed in 
parts and not as a whole. 

2. Type Style . Any simple or sans serif typeface is 
satisfactory for instructional materials. The following 
considerations are useful in deciding on the type style to 
use: 

• Bold letters are more difficult to read. 

• Letters with a fine stroke width tend to be less 
legible than letters with a medium stroke width. 

• Condensed type should be avoided. 

• Capitalized sentences are more difficult to read than 
a line combining upper and lower case letters. 

• Use of one type style throughout a document increases 
readability. 

• Numbers should not be written out, but shown as 
numbers. 

3. Spacing . The third factor to be considered in designing 
and evaluating legible text is spacing. The guidelines for 
acceptable spacing practices include: 

• Spacing between words should be consistent. 

• Leaving a space between lines increases readability. 

• Uso spaces between paragraphs to make them stand 

O'Ji. 



72 



• Indenting the first word of a paragraph is not 
necessary. 

4. Contrast. Maintain a high brightness contrast between 
print and background. Avoid dark-colored papers. Also 
avoid light letters on a dark background. 

5. Line Langth and Pago I angth The optimal line will 
contain about seven words and be between 26 and 52 
characters, with 39 optimum. Page length of 55 lines is 
oftentimes considered ideal. Right justification is not 
necessary, and may reduce readability for poor readers 
(Burbank & Pett, 1986, p. 6). 

The fourth factor regarding printed materials is the use of 
illustrations. Because the attention of the reader is focused on the 
drawings, graphs, diagrams, or photographs, care should be taken to 
ensure that illustrations are relevant to the text. Learners tend to 
scan pictures; thus, only important details should be given. Simple 
illustrations work best, and the use of cues, such as arrows, to 
focus attention on critical points is helpful. 

The final factor in creating and evaluating written 
instructional materials concerns page organization. A number of 
guidelines should be followed to assist the reader in comprehending 
the material: 

1. The structure of the document should be consistent. 

2. Logical order is critical and should be apparent to the 
reader. 

3. Different ideas should be clearly separated. The oxact 
format is less important than the ease with which 
learners can find information. 

4. Long documents require a table of contents. (Burbank & 
Pett, 1986, p. 6) 

The choice of page size and layout depends on many factors: 

the audience, the objectives, how and where the material will be 

used, and the cost. No matter what page size or layout is chosen, the 
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pages should be attractive and easy to read. The following 
guidelines may be helpful: 

1. Use numbers, headings, bullets, indenting, spacing, or 
other graphic devices to lead the reader through the 
material in a logical order. 

2. Make effective use of white space. 

3. Assure that illustrations are visually related to the text. 

4. A page should be visually balanced in order to appear 
attractive. 

5. Maintain simplicity and avoid distracting elements. 
(Burbank & Pett, 1986, p. 6) 

By following design principles, the trainer can increase the 

effectiveness of printed instructional materials. Because design 

principals do not represent an exact science, it is important to 

evaluate the materials during the formative evaluation stage 

(Burbank and Pett, 1986). Evaluation at this stage not only keeps the 

costs of making changes down, it adds to the effectiveness of the 

prcgram. 

What materials are required to make a program package 
complete? Typically, objective-based training program packages 
consist of four components: 

1. A set of instructions for trainees, 

2. An instructor's manual or guide, 

3. A trainee workbook, and 

4. Tests for each unit and/or lesson. (Dick & Casey, 1985) 
Variables affecting the scope of each of these components include 
the following considerations: 

1. To what degree is the program cirected toward 
individualized and group-oriented presentation? 
Individualized programs, such as programmed instruction 
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or self-study packages, require considerably more detail 
and anticipation of nearly every problem, issue, or 
concern that might be encountered by the user. At the 
other extreme, group-oriented instruction may consist of 
an outline or series of questions. Probably the most 
versatile format is the verbatim script, which can be 
used in either individual or group situations. 
To what degree is the program organization specific, 
that is, directed at the unique conditions found in only 
one organization, work group, job class, or position? 
Several disadvantages are associated with highly 
organization-specific programs: (a) they may be easily 
outdated as changes occur; (b) they may req jire a 
custom-design development process; or (c) tiey may need 
to be developed by in-house staff, resulting in 
potentially higher costs. At the other extreme is 
generalized education. A major disadvantage of more 
generalized programs is the increased difficulty in 
transferring the learning back to the job. 
To what degree the program will be modified before or 
during each offering determines how comprehensive the 
program package must be. At one end of this continuum 
is material that ic relevant to a single presentation; at 
the other end is a do-it-yourself kit, including needs 
assessment, course script, unit and lesson plans, visual 
aids, exercises, and the ability to modify the program 
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with little advanced skills in program design and 
development. (Sredl & Rothwell, 1987, pp. 154-158) 
Each component may be considered against these three variables 
that affect the scope of the training. 

A comprehensive list of what should be included in each of the 
components could assist a buyer in assuring that every element 
required by the purchasing organization has been considered. 
Obviously, many situations do not require that element, and some 
specific situations may require that an element be omitted. A 
master list for a comprehensive package intended for group 
presentation would contain the following items: 

1. A brief description of the instructor's role in program 
delivery. 

2. The sequence of desirable activities in preparing to offer 
the program. 

3. The type of attendees for which the program is intended. 

4. The means by which needs were (or can be) assessed. 

5. The preparation of program handouts and visual aids. 

6. Mow the presentation should be practiced. 

7. Notes on room arrangement. 

8. A checklist of equipment needed to offer the prog am. 
8. Tests or other means by which to assess stuclenl 

performance. 

10. Questionpiire{s) for use by participants in assessing 
instructor performance. 

11. A text or script. 

12. Transparencies or other visual aids needed for the 
presentation. 

13. Copies of <rr handouts, exercises, trainee workbooks, and 
oth&r program materials-including suggested solutions 
to exercises and tests. 

14. Information about how the program has been revised--or 
recommendations on how it should be revised-and about 
trends in past trainee performance. (Sredl & Rothwell, 
1987, pp.1 58-1 59) 




A master list containing the minimal requirements for trainee 
workbooks would induce the following: 

• A statement of program title and purpose 

• A summary of program objectives 

• A program outline 

• Copies of readings, handouts, and exercises 

• Copies of important visual aids 

• Copies of supplementary reading or additional 
information that might be of interest to participants 

• Copies of any organizational policies and/or procedures 
related to program content (Sredl & Rothwell, 1987, 
pp.160-161) 

Individualized workbooks would require additional information and 
also should include some type of test. 

Organizational Infiuences on Training 
An important aspect of any decision to purchase a vendor- 
supplied training program is the consideration of organizational 
factors. For example, McAllaster (1987) emphasized the importance 
of management commitment to the training effort. Strother and 
Klus (1982. pp. 45-46) described the individuals who are responsible 
for decision making as: 

constrained by available channels of information, 
organizational routines, and the physical limitations and 
biases of the people on whom they depend.... Administration 
controls can also be factors.... These constraints are 
internalized as organizational givens and decisions must be 
made within this framework, (citec* in McAllaster, 1987, p. 60) 

Perhaps the most important organizational factor to consider 

in making decisions concerning training programs is the 

consideration of corporate culture. Schein (1984) defined 

organizational nulture as: 
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the pattern of basic assumptions that a given group has 
invented, discovered, or developed in learning to cope with its 
problems of external adaptation and internal integration and 
that have worked well enough to be considered valid, and 
therefore, to be taught to ;oW members as the correct way to 
perceive, think, and feel in relation to those problems (p. 3). 

Allen and Silverzweig (cited in Craig. 1976) discussed the 

importance of understanding culture and its impact on participant 

receptivity to, and the subsequent effectiveness of, training efforts. 

The underlying assumptions in the organization determine its norms, 

and norms reflect the behavior associated with how the underlying 

assumptions are lived out in an organization. They are influGnced by 

the following elements: 

1. Leadership commitment-visible, verbal, and active 
support of behaviors. If a program is attempting to 
instill a new concept or management principle (such as 
teamwork), it must be supported by the organizational 
leadership. 

2. Modeling behavior-prestigious members often serve as 
role models. Behaviors they exhibit will become readily 
accepted. 

3. Information feedback-regular feedback reinforces 
norms. When information on a particular norm starts to 
diminish, it tends to become less important to members 
of the organization. 

4. Secognition and reward-behavior that is rewarded will 
be repeated; if the reward continues over time, it 
becomes a norm. 

5. Knowledge and skill development-lack of ability in 
demonstrating the skills or behaviors associated with a 
norm will cause it to diminish (for example, a supervisor 
who lacks skills in performance appraisal may not 
provide the appropriate recognition). 

6. Orientation-the most teachable time for employees is 
when they are new to a position or company. New 
employee or promotion orientations that are left to occur 
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as an unstructured experience may expose the individual 
to norms that are opposite to organizational objectives. 
7. Supervisory follow-through-supervisors are a key 

element to effective norm-maintenance or norm-change 
programs. What a supervisor supports is likely to be 
reinforced in the organization. (Allen & Silverzweig cited 
in Craig, 1976, pp. 12-3 to 17-5) 

Training that is consistent with the organization's norms will be 

encouraged and rewarded, but training that violates norms will be 

confronted. The more the objectives of a training program run 

counter to tho existing norms, the more critical it is that the above 

elements be considered as part of the change effort. 

The degree of compatibility of the training with the culture is 
an essential factor in the assessment and analysis processes in the 
initial stages of determining the training need. If the existing 
norms are clearly understood and the norms that are relevant to the 
training need are clearly identified, the likelihood that the decision 
maker will be abh to consider the cultural implications of 
purchasing a training program is increased. However, rarely are the 
values and norms of a particular training program explicit. Thus, 
skill in identifying the underlying values and norms inherent in a 
program is essential (see E. H. Schein, Organizational Culture and 
Leadership. Chapter 5: How to Uncover Cultural Assumptions in an 
Organization, 1985). 

Four additional organizational factors that affect training 
were identified by Killian (1976): 

1. Who is responsible and accountable in the organization to 
insure that training and development occurs; 

2. What resources are made available or unavailable for 
training (budgets, trainees' time, etc.); 
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3. How training an J development activities are integrated 
into the company's philosophy, goals, and operating 
practices; and 

4. How decisions are made in the company that relate to 
what training activities are run. (pp. 111-113) 

In addition, Nadler (1981) discussed several other factors that 

affect the direction, effectiveness, organk^ation, and type of 

training programs held within a company. They include: 

1. Who has budget control of training funds; 

2. How attendance at training programs is determined; 

3. Who is responsible for ensuring that learning is 
transferred from the class to the job; and 

4. Who expects to see the results from training, (cited in 
McAllaste^ 1987, pp. 63-64) 

And finally, the following organizational factors that 

contribute to the failure of training efforts were listed by Spitzer 



(1984): 




1. 


Training viewed as a fringe benefit; 


2. 


Training as a quick fix for organizational problems; 


3. 


Delegation of training responsibility from managers and 




supervisors to the training function; 


4. 


Lack of management commitment; 


5. 


Reluctance to hold training departments accountable; 


6. 


Failure to identify the rea' training needs; 


7. 


Lack of aids to transfer learning back onto the job; 


8. 


Inappropriate trainees; 


9. 


Lack of opportunity to use now skills on the job; and 


10. 


Lack of follow-up after training, (pp. 6-10) 



McAllaster (1987) offered the follovWng summary of the 
organizational factors associated with training: 



1. Internal controls on how training is administered; 

2. How the "normative" system of the organization affects 
training; 
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3. How politics affects the training function and 
programing; 

4. How the daily activities of the company affect training 
programming; 

5. Constraints and barriers to training and development 
within the company; 

6. The reputation of training in the company; 

7. Whose responsibility it is to see that employees are 
adequately trained; 

8. If the training department is held accountable for the 
programs it runs; 

9. How and what determines the types of programs run 
within a company; 

10. How attendance at programs is determined; 

11. How the transfer of learning is insured in the company; 
and 

12. How the future of the organization is expected to impact 
the organization, (pp. 68-69) 

The following could be added to the above list: 

13: How management commitment to the training effort is 
obtained and maintained. 

Summary of the Literature Review 

The factors associated with the decision to build a program 
in-house or to purchase a program from an outside source were 
considered and various sources that discussed the potential 
selection factors were reviewed. Sources in which external 
programs could be found or purchased were given. Finally, the 
factors associated with instructional development systems, training 
evaluation, instructional design and delivery, and organizational 
influences on training were described. This literature will serve as 
the basis of the model and instrument described in chapter 4. 
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CHAPTER THREE - METHODOLOGY 

The methodology of this paper was determined by the need to 
provide a valid basis for developing and substantiating a program 
selection model and instrument. Future research that will be 
• equired to substantiate the model in practice remains beyond the 
focus of the present study. 

Investigation Method 

Because of the lack of research concerning the purchase of 
vendor-supplied programs, this topic will be developed 
theoretically. The model and instrument provided here are based on 
training theory that was revealed through a thorough review of the 
literature. Where research from related fields or subjects is 
applicable, appropriate use of that data is cited. 

Research Basis 

This investigation is based in the field of educational 
evaluation. Educational flvaluation is defined by Borg and Gall 
(1983) as "the process of making judgments about the merit, value, 
or worth of educational programs, projects, materials, and 
techniques" (p. 733). Because the factors involved in the 
investigative process used to create the model and instrument for 
this study are similar to the factors involved in a decision maker's 
implementation of the model and instrument; they will be delineated 
in considerable detail. 

The field of educational evaluation has provided important 
tools for policy analysis and development, the political decision- 
making process, and program management. Costs, benefits, and 
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problems of various program alternatives are prepared in the form 
of position papers for decision-makers to review during policy 
analysis. Increasingly, evaluation findings are being used by 
politicians in the political process to create support for or lo 
advocate cuts in various educational programs. As a project 
management tool, evaluation research is used to determine costs, 
benefits, and efficiency ratings, so thai managers can be held 
accountable for producing results. It is also used to generate data 
that will assist managers in making sound decisions regarding 
program design, personnel, and budgets (Borg & Gall, 1983). Thus, 
educational research has much in common with training evaluation 
and research. 

Because of their importance in educational research, standards 
have been developed by which to judge the quality of educational 
evaluations. The Joint Committee on Standards for Educational 
Evaluation (1981) published criteria for a good evaluation study. 
The criteria ate based on utility, feasibility, propriety, and 
accuracy. 

An evaluation has utility if it is informative, timely, and 
useful to the affeclsd persons. Feasibility means, first, that 
the evaluation desigri is appropriate to tho setting in which the 
study is to be conducted, and second, that the design is cost- 
effective. An evaluation has propriety if the rights of persons 
affected by the evaluation are protected. Finally, accuracy 
refers to the extent to which an evaluation study has produced 
vJid, reliable, and comprehensive information about the entity 
being evaluated. (Borg & Gall, 1983, p. 739) 

Thirty standards were established by which to operationalize 

each criterion. Borg and Gall (1983) created a list of the standards 
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regarding the criterion to which each standard is most closely 
related. Their list is useful as the basis by which (a) to evaluate the 
results of the present study, (b) to judge the soundness of previous 
evaluation models, and (c) to provide guidance for the individuals 
who implement the model and instrument developed here. These 
standards are listed below: 

Utility 

1 . Audience identification. All of the audiences affected by 
the evaluation should be identified. 

2. ^valuator credibility . The evaluator should be competent 
and trustworthy. 

3. informatio n scope and selection . The questions to be 
answered by the evaluation should be pertinent and 
responsive to the affected audiences. 

4. Valuation interpretation . The bases for interpreting the 
results and for making value judgments should be clearly 
described. 

5. Report clarity. The affected audiences should find it 
easy to understand the evaluators' reports. 

6. Report di ssemination . Evaluation reports should be 
disseminated to all clients and right-to-know audiences. 

7. Reoort timeliness . The evaluation findings should be 
reported in a timely manner. 

8. Evaluation impact . The evaluation should be conducted so 
as to encourage appropriate action by the affected 
audiences. 

Feasibility 

9. Practical procedures . The evaluation procedures should 
be practical and minimall/ disruptive to participants. 

10. Political viability . The evaluators should obtain the 
cooperation of affected interest groups and should keep 
any group from subverting the evaluation process. 

1 1 . Cost effectiveness . The benefits produced by the 
evaluation should justify the resources expended on it. 
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Propriety 



12. Formal nhiigatinn Obligations of all involved parties 
should be agreed to in writing. 

13. Conflict of interest Conflicts that arise in the 
evaluation process should be treated openly and honestly. 

14. Full and frank disclosure. Evaluation reports should be 
direct and honest. 

15. Public's right to know. The public's right to know about 
the evaluation should be assured whenever legally or 
ethically permissible. 

16. Rights of human suhjflcts The rights and welfare of 
persons involved ir< the evaluation should be protected. 

17. Human interactions. Evaluators should respect the worth 
and dignity of persons involved in the study. 

18. Balanced reporting. The strengths and weaknesses of the 
entity being evaluated should be reported completely and 
fairly. 

19- Fiscal responsibility. Expenditure of resources for the 
evaluation should be prudent and ethically responsible. 

Accuracy 

20. Object identification All pertinent aspects of the entity 
being evaluated should be described. 

21 . Context analysis All pertinent aspects of the conditions 
that surround the entity being evaluated should be 
described. 

22. Described purposes a nd procedures . A careful record of 
the evaluation purposes and procedures should be kept. 

23. Defensible information sources Sources of data should 
be described in sufficient detail that their adequacy can 
be judged. 

24. Valid measurement A range of validated measures 
should be used in the data collection process. 

25. Reliable measurement. The measures should have 
adequate reliability for their intended uses. 

26. Systematic data contro! Human error in data collection 
should be minimized. 

27. Analysis of quantitati ve information . Analysis of 
quantitative data in an evaluation study should be 
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accurate and thorough, and should yield clear 
interpretations. 

28. Analysis of qualitative information. Analysis of 
qualitative data In an evaluation study should be 
accurate and thorough, and should yield clear 
interpretations. 

29. Justified conclusions . The conclusions of an evaluation 
must be based on sound logic and appropriate data 
analyses. 

30. Objective reporting . Evaluation reports should be 
thorough and free of biases of pressure groups. (Borg & 
Gall, 1983, pp. 739-741) 

Each standard is explained and illustrated by case studies in the 

Joint Committee's report. Clearly, the importance of each standard 

wilt change according to the situation. For example, a business may 

not consider the public's right to know as important or even 

relevant, whereas a public agency would. 

Besides the failure to consider the above standards, 

sometimes other mistakes are made when evaluation research is 

conducted. The following list of evaluation errors will serve as a 

part of Ihe basis by which to judge this investigation. The 

evatuator: 

1. Ignores some standards relating to the utility, 
feasibility, propriety, and accuracy in designing an 
evaluation study (as mentioned before). 

2. Fails to deflnaate all aspects of the program that is 
being evaluated. 

3. Does not use measures that are directly linked to 
program goals. 

4. Ignores possible side-effects not included in the forma! 
statement of program goals. 

5. Does not relate evaluation findings to decisions that 
need to be made about the program. 

6. Does not consider alternative models of evaluation In 
designing a study. 
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7. Does not consider using both qualitative and quantitative 
instruments in designing a formative or summative 
evaluation. (Borg & Gall, 1983, p. 766) 

Study Design 

There were seven steps in this investigation. Step One was a 
review of the literature. This study was conducted to determine the 
criteria previously identified by experts and authorities as 
important to the vendor program selection decision. The literature 
search was expanded to include a review of related areas. A review 
of instructional development systems and program components 
assisted in determining key criteria in the development cycle, which 
could be evaluated before the program was implemented or pilot 
tested. Literature concerning training evaluation, the decision to 
build a program in-house rather than to purchase from an outside 
source, and sources of externally supplied programs were also 
reviewed. This activity resulted in a set of criteria that could be 
used during the selection process. 

Step Two was to develop a simplified system by which a 
decision maker could make logical sense out of the identified 
criteria. After an analysis of the information derived from the 
literature search, the criteria were synthesized into four 
dimensions. EsserTtially, this synthesis occurred out of a creative 
process. If the criteria found in the literature could be logically 
mapped and accounted for by the model, the initial face validity of 
the mode! would be established. 

To operationalize the model. Step Three required taking each 
criterion and operationally defining it based on a summary of the 
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applicable material found in the research and literature. In addition, 
information related to the importance of the criterion comparea lo 
other criteria was noted. Based on the importance of the criteria to 
the evaluator's particular situation, a weight was assigned each 
criterion. Receiving a low rating on a heavily weighted criterion 
might disqualify a program from any further consideration. 

In Step Four, the critical information for each criterion was 
summarized and transferred onto an instrument. Worksheets were 
created to assist decision makers in reviewing a program, or set of 
programs, against each criterion identified in the model. The 
worksheet format included only as much information as would be 
required for a decision maker to make valid ratings of each program 
for each criterion. 

An overview of the model and instrument are given in Figure 2. 
At the first position in the figure, the model is shown. At the 
second position, the model has been developed into a criteria 
selection checklist from which the program reviewer can select the 
criteria he or sha is going to use in evaluating the programs. After 
selecting the criteria, a relative weight based on the importance of 
each criterion to the decision-making process is assigned. The third 
position in Figure 2 shows the worksheets that were developed to 
assist the reviewer in evaluating each of the selected criteria. For 
every criterion listed on the criteria selection checklist, there is a 
corresponding worksheet. The fourth position shows a summary 
worksheet, where data from the previously completed worksheets is 
pulled together. The ratings from previous worksheets are listed in 
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the appropriate column, multiplied by the weighting, and summed to 
achieve an overall rating for each program. 





Figure 2 --Qverview of the Methodology 

The purpose of this last position is to collect the data on each 
worksheet and make it into a decision matrix. This step requires 
that each criterion be scored according to a common scale, even 
though the criteria may be measured by different methods. The 
result of each worksheet is a rating of each program on each 
criterion considered to be important by the decision maker. For 
example, if four programs are compared, the program is assigned a 
rating scale from 1 to 4 for each criterion; 4.0 designates that the 
program met all requirements in an extraordinary manner, and a 0.0 
designates that the program did not meet even minimal 
requirements. The ratings are then multiplied by the previously 
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assigned weights. These figures are summed and an overall rating 
for each program is reached. 

In Step Five the model and instrument were analyzed for their 
strengths, weaknesses, and limitations. Finally, conclusion.*! are 
presented and recommendations for further research are given. 

Limitations 

Limitations of this model include application, scope, and 
validity issues. Because the focus of this paper is theoretical, 
limitations from a practical approach are not considered. This paper 
only represents a theoretical attempt, based on the literature, to 
determine selection criteria and methods for rating programs 
against these criteria. The next stage of evaluation, which would 
represent a more practical focus, would follow one of several 
differv3nt avenues. The instrument could be tested by several 
practitione.''s in real-world situations. A study is needed to 
determine more accurately how the decision-making process is 
presently performed. Strengths and weaknesses of buying and using 
vendor-supplied training programs could be further researched. This 
practical test of the modei and tne instrument is beyond the present 
study. 

The validity of the instrument and model could be assessed by 
comparing decisions made both with and without the use of the 
instrument. Other methods of program selection might be identified 
and compared with the model and instrument. The predictive and 
concurrent validity issues associated with the instrument will not 
be addressed in this investigation. 
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No information will be provided concerning the reliability of 
the instrument and model. Reliability could be assessed by first 
asking several training decision makers to use the instrument on the 
same set of programs and then by comparing their results. Another 
limitation of this study concerns the degree to which the model uses 
each of the criteria that are important to the selection process. 
This study will not assess content validity. The scope of this 
investigation will not include determining whether the study has 
identified an exhaustive list of criteria. 

In addition to not testing whether the listing is exhaustive, 
very little will be discovered about the relative importance of the 
criteria. Are there some criteria that should be considered in every 
situation and other criteria that have little or no bearing on an 
individual program's effectiveness? This question will not be 
answered in this study. 

Another limitation of the study concerns the availability of 
the information required to use the instrument. Will vendors be 
willing to provide decision makers with access to the information 
they need in order to perform this type of evaluation? This issue 
also will not be addressed. 
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CHAPTER FOUR ~ SYNTHESIS 

Before the criteria discussed in the literature review can be 
synthesized into a model and instrument for the selection of vendor- 
supplied programs, an overview framework to guide the synthesis 
discussion is needed. A logical approach would be to ask what steps 
would an individual or team follow, consciously or unconsciously, 
when making a decision to purchase a training program? 

From a review of models concerning how people go about 
purchasing products and services, such a flow chart was made. 
Stanton (1987) provided a representative model of the purchasing 
process. When an aroused need or want occurs in an individual or 
group, actions are taken to satisfy that need or motive. The course 
taken is determined by the buyer's perceptions. The elements that 
influence the buyer's perceptions and frame of reference include 
cultural and psychological aspects. The cultural aspects that 
influence the buyer's perception include the culture, social class and 
social group, small reference groups, and family. The psychological 
aspects include learning experiences, personality attitudes and 
beliefs, and self-concept. All of these factors interact in the 
formulation of the consi'mer's perceptions, which influences ouying 
behavior. 

The buying benavior is identified as a five-step problem- 
solving approach: 

1. Recognition of an unsatisfied need 

2. Identification of alternative ways of reducing tensions 
(i.e., achieving satisfaction) 

3. Evaluation of alternat'ves 
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4. Purchase decision 

5. Postpurclnase belnavior (Stanton, 1987, p. 115) 

Tine buying, decision-making process is based on Dewey's rational 
problem-solving model (referenced in Conrad, 1985): 

1. Set or determine a goal, based on a problem or situation 
requiring strategy; 

2. Search for information relevant to the problem or 
strategy; 

3. Develop alternatives to be considered in solving the 
problem or strategy; 

4. Weigh the pros and cons of alternatives under 
consideration; 

5. Choose the best alternative; and 

6. Take action. 

Dewey's rational individual process was applied to organizations' 
results in the following rational-actor model of organizational 
decision making: 

1. Individual employees rationally assess goals, options, 
and possible gains and losses. 

2. Individual employees participate in decision-making 
events, present varyir.^ viewpoints and expert 
information, and cooperate in making a rational decision. 

3. Organizational decisions are made. 

4. Decisions have effects which are known and understood 
by all parties, who store this information to be used in 
future decisions. (Conrad, 1985, p. 148) 

The rational models cannot be relied upon because people and 

organizations are not totally rational. The manner in which people 

actually make decisions does not follow this process. Studies 

Indicate that humans do not behave as rational actors, and, at best, 
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they only appear to act in ways consistent with this model (Conrad, 
1»85). 

Thi question then becomes: How do people in organizations 
actually make decisions? 

Observations of actual human decision making, including 
organizational decision making, suggest that often we reverse 
the sequence, first making choices and acting on them and then 
seeking out the information and beliefs which will support 
them. We discover, for instance, that we have married a 
shortstop, and then determine that we value excitement and 
passion more than security. We make a decision that seems to 
be a correct one and then construct a picture of our decision- 
making process that makes us seem to be rational (Conrad, 
1985, p. 153). 

Various contingency theories of why ana how people actually act in 
ways that are inconsistent with the myth of rationality have been 
presented. Karl Weick's model accounted for the nonrational cr the 
limited or bour^ded rationalily of people and systems by explaining 
that people first act or make a decision (enactment), next observe 
what it is that happened (selection), and finally construct an 
explajiation of the action or decision (retention) (Weick, 1979). The 
garbage can theory of ritualized decision-making processes 
presented by March and his associates (Cohen, 1972) challenges the 
belief that the main focus of decision making is solving problems. 
The focus on solving problems often is compromised by the need to 
take action. 

Whatever the focus, there is a need is to make complex 
problems more manageable. Conrad identifies three different styles 
by which decisions can be made: (a) oversight is to make a quick 
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decision because a problem will become even more complex if 
nothing is done; (b) flight involves avoiding making a decision until 
other people act, thereby reducing the complexity of the problem ; 
and (c) resolution approximates the rational actor model discussed 
previously (Conrad. 1985). Thus, the decision-making process is 
determined, in paa, by the complexity of the decision in question, by 
the likelihood of creating a favorable outcome, and by other 
organizational factors such as control mechanisms and history. 
Simple situations in which most of the information required to make 
a decision is readily available and finite in nature require a 
different problem-solving process than complex, ambiguous 
situations, in which totally rational decision making ic impossible 
(Conrad, 1985). 

Alan Meyer (1977) gave an exampio of how the rational and 
nonrational aspects of decision making occur in organizations in his 
study of how hospitals make capital-equipment purchasing 
decisions. The decision making process began with an orderly 
consideration of the various objective aspects involved in making 
the decision: determining program needs, equipment costs, and 
projected payoff periods. 

In many of the episodes, the actions started to deviate from 
the rational model. Communication among participants became 
more vague and imprecise and started to focus on abstract and 
intangible topics, like the parties' shared beliefs, values, goals 
for ihe hospital and vision of its future. Later the decision 
makers began to restructure and redefine what actually had 
taken place during the deliberations so that they seemed to fit 
the myths of rational decision making. (Conrad, 1985, p. 163) 



Two important purposes were served by this dual process. First, the 
members of the selection team were able to view themselves as 
tough but cooperative members of a functioning group. Second, the 
symbolic aspects of the nonrational methods of the decision-making 
process allowed them to gam a psychological closure on the process 
and a sense of unification as a group. In situations in which the 
focus remained on rational methods throughout the process, "the 
group seldom reunified. Dissension continued, debates proliferated, 
and in some cases key staff members resigned and expensive new 
equipment was left sitting in the basement" (Conrad, 1985, p. 163). 

Meyer's study shows that rational and nonrational processes 
serve decision makers in different ways. No matter how complex or 
ambiguous the decision, organizations do develop 'successful 
patterns of communications. But, two potential problems exist. 
First, no ideal model of communication or decision making can be 
imposed on an organization. Rather, strategies will only succeed 
"where they are appropriate, respect, and adjust to the complex 
patterns of action which have emerged in a particular organization" 
(Conrad, 1985, p. 164). Second, organizations can become trapped in 
their successful patterns, and the flexibility and responsiveness to 
handle new situations may be diminished. An increased awareness 
of the communication patterns that have developed, a commitment 
to discovering new strategies to increase adaptability, and the 
ability to obtain and process the kind of information that casts 
doubt on perceptions, beliefs, and interpretations inherent in the 
system are needed to avoid these potential problems. 
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Following this discuss'on of purchasing behavior and decision- 
making processes, certain factors will need to be accounted 'or. In 
addition to the rational elements of the decision-making process 
that are assumed in the use of a model and instrument, allowances 
must be made for organizational factors. Also, deviations from the 
rational model are expected in most practical applications of the 
model and instrument. 

Based on these considerations, the following flow chart 
illustrates a typical decision-making process for selecting a 
vendor-supplied program. 
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Figure 3 --Flowchart of Proposed Program Selection Process 



Three sets of forces are identified as interacting to influence 
the perceptions by which the training program buyer makes a 
decision: external, intraorganizational, and individual. 
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1 Individual forces include aspects associated with the personal 
and psychological characteristics of participants in the purchasing 
process. 

2. Intraorganizational forces are associated with the nature of 
the organization and its influences (such as its culturo, structure, 
and policies) on the purchasing process. 

3. External forces are factors in the environment and society, 
such as outside reference groups and government regulations, that 
affect the purchasing process. 

These three sets of forces work together in determining the 
perceptions and interactions of the buyer and seller in the sales 
process. 

Learning relates to the previous buying experiences. Based on 
an individual's or organization's previous experiences, habitual 
responses are created that determine future buying patterns. 
Organizational purchases differ from consumer purchases because 
items such as training may not involve much trial-and-error 
decision making. The magnitude of such decisions makes the trial- 
and-error approach much too costly, and it is much less likely that 
it will be used. 

Personality refers to the hrbitual patterns of traits that 
people develop that partly determine their actions. In situations 
involving face-to-face selling, as in selling and purchasing many 
training programs, the personalities of the people involved will be a 
key factor. How the salesperson's personality is perceived by the 
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purchasing organization will have a major effect on the purchasing 
decision. 

Although organizational buyers tend to consider more 
objective criteria than do individual consumers, their attitudes and 
beliefs can still play an important role in the organizational 
decision-making process. Attitudes and beliefs refer to the long- 
term value judgments, feelings, and opinions that buyers hold about 
a product or service. These are formed from past learning 
experiences and relationships within reference groups such z.s work 
associates and family. For organizational purchases, factors 
associated with the vendor, (i.e., length of time in business) rather 
than those directly associated with the product (i.e., speed of 
delivery), play a larger role than in consumer purchases (Busch & 
Houston, 1985). 

How thtf individual and organization see themselves is an 
important factor in the purchasing process. The self-concept or 
self-image includes both the actual way one sees oneself and the 
ideal way one would like to see oneself. A vendor thai presents to 
the buyer in a manner that is consistent with the decision maker's or 
organization's self-concept stands a much higher chance of being 
delected. 

Motives and commitments are based, in part, on the needs of 
the individual or organization. Products identified as consistent 
with the buyer's commitments and motives-as in cost reduction, 
return on investment, risk reduction, or increasing individual 
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status-contribute to the development of perceptions that influence 
behaviors in the buying process. 

Another organizational factor influencing the purchasing 
process is culture. Culture includes the symbols, artifacts, and 
assumptions created by a given group and handed down from member 
io member as determinants and regulators of behavior. The culture 
includes intangible items, such as attitudes, beliefs, values, or 
languages, and tangible items, such as products, tools, or buildings. 
Culture can be conscious or unconscious. For the most part, it is a 
powerful and unconscious determinant of the cultural members' 
actions and thoughts. Likewise, the purchasing process is subject to 
the powerful, and for the most part, unconscious influences of the 
organizational culture. An organization develops a personality and « 
self-concept about what is acceptable behavior. For example, just 
as an older consumer would often not consider buying younger, more 
modern clothing, an organization that vi',^ws itself as established 
and conservative would often not consider choosing flashy, modern 
training programs. Norms, policies, and structures operate to 
determine who makes the initial purchase decision, who approves 
the decision, who ib responsible for implementing the decision, and 
other factors associated with ourchasing and implementing a 
training program. 

Where the organization is headsd in terms of its stated 
mission, goals, visions, and strategies may be a powerful influence 
on the purchasing process. A company that is attempting to change 
directions and cope with massive change may consider alternatives 
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different from those that would be indicated by its present culture 
and self-concept. 

Group processes play an important role in how team or 
committee selection decisions are made. Groupthlnk . or the process 
by which groups develop a desperatu need tc reach consensus at any 
cost, is one factor among many that may operate to undermine a 
group's decision-making processes (Janis, 1971). 

In addition to the organizational and individual forces, 
external forces influence the formulation of decision -making 
perceptions. Worldwide competition and government actions 
represent large-scale external forces that influence the purchasing 
process. A reference group refers to the group of people who 
influence a person's «:ittitudes, values, and behavior by es':.blishing 
normative behavior patterns. These serve as a frame of reference to 
which the individual is expected to conform. Thus, the reference 
groups with which individuals and organizations identify provide 
smaller-scale influences on the purchasing process. Sociccultural 
factors influence individuals and organizations based on such 
sources as family, church, school, and language. Patterns, 
expectations, and norms may be influenced by such factors as 
amount of education, socioeconomic status, and shared values. 

In summary, individual, organizational, and external forces 
influence the formulation of decision-making perceptions. Buying 
behaviors are based on these perceptions, and these perceptions have 
an impact on the training program selection process. 
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Not all selection criteria will have the same impact on the 
purchr^se decision, which means that the criteria must be weighted 
in some manner. Certain criteria may be critical: If the program is 
unable to satisfy such criteria, it will not be considered. Other 
criteria may be less critical, or even optional, to the program's 
effectiveness. If these less critical criterial are included in a 
particular program, it may be rated higher than programs not 
including them. 

Outside suppliers of programs must be identified and 
contacted for needed information. Based on the criteria selected, 
the program supplier will be asked specific questions about the 
program and requested to supply information sufficient to fill out 
the worksheet questions associated with the selected criteria. 
After the individual worksheets have been filled out, the data are 
transferred to a master worksheet to determine the highest rated 
program. If none of the programs qualifies by meeting the minimum 
requirements, the best of the nonqualifying programs may need to be 
customized in-house to meet the selected criteria. 

At this stage, the selected program can be pilot-tested. If the 
program meets the requirements in actual practice, it can be 
purchased and implemented. It should be noted that meeting the 
criteria selected thus far does not imply that the program will be 
effective. Although the evaluation processes up to this point are 
designed to increase the probability of selecting the most effective 
and efficient program, there is no guarantee that this will be so. 

Criteria in Four Dimensions 
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Certain aspects of the training program selection process 
were discussed in the literature review. Sources of externally- 
produced programs and factors associated with the build in-house or 
buy-outside decision also were discussed. The next critical steps 
are selecting the criteria, weighting the criteria, filling out the 
worksheets, summarizing the information on a final worksheet, and, 
finally, making a decision. 

The factors and criteria identified in the literature as 
important to evaluating a vendor-supplied program can be clustered 
around four dimensions: 

1. Contftnt Factors . How well does the content of the program 
fit the identified training needs of the purchasing organization? 

2. Design Factors . How well are solid instructional design and 
delivery methods utilized in the program? Are they consistent with 
the program content? Do they fit the requirements of the purchasing 
organization? 

3. Coptrol Factors . How much research went into the 
development of the program? Has it been evaluated for 
effectiveness? 

4. Context Factors . How well does the program fit the 
administrative aspects of the purchasing organization? How does 
the vendor's company or organization rate in being able to meet the 
needs of the buyer? 

The review of instructional development systems follows the 
same four dimensions. The Training Technology System contained 
phases that were associated with the content dimension (analysis 
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phase), design dimension (design and develop phases), context 
dimension (implement phase), and control dimension (control phase). 
The matching of the content of the program with the needs of the 
purchasing organization requires a competent needs assessment and 
analysis in order to provide the data needed to make the 
content/neads comparison. The design and develop phases include 
the program design, lesson design, lesson plan, and training 
materials development. The program management plan and factors 
involved in the delivery of the training are associated with the 
context dimension. The control phase emphasizes training 
evaluation, effectiveness evaluation, and follow-up training, which 
are associated witl. the control dimension. 

The review of the literature on instructional design and 
delivery revealed a list of criteria to consider in the design 
dimension. Instructional system factors of objectives, content 
structuring, instructional sequencing, and adherence to principles of 
adult learning and the implementation factors of rate of delivery, 
repetition and practice, motivation, reinforcement and rewards, 
presentation methods, media planning, and instructional materials 
appear to meet the requirement of the design dimension. Other 
factors associated with testing and measuring performance and 
transfer of learning to the job could be considered part of both the 
control dimension and the design dimension. Because the ultimate 
value of each of these factors deals more directly with training 
effectiveness and evaluation, they will be considered from the 
perspective of the control dimension. Trainee characteristics and 
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trainee readiness are compared on tine basis of program intent. TInis 
comparison is similar to the content/'needs comparison and will be 
considered in the content dimension, even though trainee 
characteristics and trainee readiness also are important to design. 

Context, in this sense, refers to the criteria, not considered in 
other dimensions, that are important to the selection process. The 
discussion of organizational variables as well as the organizational 
influences on the purchasing decision-making process are important 
issues in the context dimension. In addition, issues in the vendor's 
organization are also critical to the selection process. These 
issues, as identified in the literature concerning the build in-house 
or purchase outside decision-making process, also will be included 
in the context dimension. 

This information may be summarized in the following 
selection criteria model: 
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PROGRAM SELECTION MODEL 



What content needs to be 
trained tc whom? 



CONTENT I 




How well is the 
program designed? 



£iaii£fi_4--Program Selection Model 

Several of the models reviewed earlier advocated certain processes 
by which the purchasing organization could better evaluate each of 
the factors that are important to the decision-making process. This 
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information could serve as the basis for developing the model into 
an instrument that can be used by decision makers to select vendor- 
supplied training programs. 

Development of the Training Program Selection Instrument 
Two of this study's objectives have now been met: (a) the 
literature associated with various aspects of the decision-making 
process used in selecting a vendor-supplied training program has 
been reviewed and (b) the organization of the criteria into the 
Training Program Selection Criteria Model has been accomplished. 
The next phase of this study is: (a) to develop a process for 
assessing each of the criteria identified and (b) to provide a method 
by which to summarize the assessed data. Pilot-testing both the 
instrument and this process to determine its validity and reliability 
will be conducted in a later study. 

To translate the model into a selection instrument, a checklist 
of the identified criteria was created: 
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PROGRAM SELECTION CRITERIA 
CHECKLIST 

Program PrApiir«il hy 




DESIGN caSSe) WBGHT 


POMTPYT TOBE 












CRITERiON 








ObiectivM 






Prioa 






Structuring th* Content 






Ability to Dalivar 






Instructional Stoutnctna 






Cultural Fit 






Adult LMming PrindolM 






Instnictor Training 






Rata of Dalvary 






Vandor Canabilitv 






Rapatitfon and Practica 






Cuttoniization 






Rainforcamant^awards 












Praaantation Mathods 












Madia Ranning 












Intmjcllonai Matarialt 






















































CRTTERION 






CRITERION 








Naads/Contant 






Tra^afar of Laaming 






Subjact MattarRaviaw 






Rafarancat Rating 






Trainaa CharactariatiGa 






Taating/Maaauramant 






Trainaa Raadlnaaa 






Naada Aaaaaamant 












Evaluation Research 




























































































CONTENT ^ 


CONTROL 





Figure 5-Training Program Selection Criteria Checklist 
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In any selection process, certain criteria may be selected as 
Important, whereas other criteria may not be given as much, if any, 
consideration. Of the criteria selected as important, some may be 
considerably more important than others. How to use the selection 
checklist and assign an appropriate weight to each of the selected 
criteria will be considered in more detail in the following sections. 
Selecting and Weighting the Criteria 
How does the reviewer determine which criteria are relevant 
to the program selection process? After selecting relevant criteria, 
how does the reviewer determine what weight to assign each 
criterion? The Joint Committee on Standards for Educational 
Evaluation (1981) stated that a high-quality evaluation was judged 
on the basis of its utility, feasibility, propriety, and accuracy. The 
basis for selecting and weighting the criteria that are important to 
the selection process must be chosen with these considerations in 
mind. 

The utility of the criteria selected is based on who will make 
use of the information generated by the evaluation process. This 
may be an individual, but most probably it will be a group of people 
who have some stake in the training program selected, that is, 
stakeholders. Only the criteria identified by the stakeholders as 
relevant and important to the decision-making process are used for 
evaluating programs. 

The utility, in many respects, is best represented as an 
evaluation design question. The key factor in evaluation design, in 
terms of the program evaluation literature, is to determine "what 
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information should be collected for what purposes to answer what 
questions" (Brini^erhoff, Brethower, Hluchyj, & Nowal<owsl<i, 1983, 
pp. 77-78). "The key is to match the type{s) of evaluation to the 
information needs of specific stal^eholders and intended information 
users" (Patton, 1986, p. 149). 

Feasibility issues are associated with how important the 
selection decision is to the organization. If a program will be used 
only once, have a small attendance, and address an issue that is not 
very important, the energy and costs associated with the evaluation 
should be kept low. Reducing the time, energy, number of criteria 
selected, and depth with which the criteria would be reviewed will 
reduce the cost of evaluation. On the other hand, for programs that 
represent a major investment on the part of the organization and 
address a critical business concern, considerably more criteria may 
be examined in greater detail. 

In addition to utility and feasibility, the issue of propriety 
must be addressed, that is, that the rights of ail persons affected by 
the evaluation are protected. For example, a study of low achievers 
in an organization would show propriety if lists of the involved 
employees were not made known to the public. In addition, accuracy 
must be maintained to assure the validity of overall process. 

Again, the weight assigned to each criterion is determined by 
the stakeholders. Certain criteria may represent nonnegotiable 
items that must be present in the selected program. Obviously, 
these killer criteria would be weighted the highest (closer to 100). 
Other criteria may not be as important, and would be appropriately 
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weighted at lesser amounts (90, 80,. ..10). Totally unimportant 
criteria would be rated 0. A weighting scale from 0 to 100 is one 
possibility, and buyers could develop other rating scales based on 
their own requirements. 

Evaluation of the Criteria in the Content Dimension 

In the content dimension, the major focus 'should be on how 
well the content of the program fits the content needs that were 
identified as important to the purchasing organization. In addition, 
the characteristics and readiness of the trainees must be compared 
with the program's assumptions concerning these two factors. The 
vendor may offer the capability of performing a needs assessment 
for the purchasing organization or may provide tools for the 
purchasing organization to perform its own needs assessment. 
Content/Needs Comparison 

The most important criterion In this dimension is the 
content/needs comparison. !f the selected program will not solve 
the training problem, no matter how exciting or low-cost, it will 
make little difference in the buyer's organization. A proposed 
worksheet to assess this comparison is as follows: 
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CONTENT/NEEDS COMPARISION WORKSHEET 



Progrom 
Date 



Prepared by 
Page 



of 



ORGANIZATION 
NEEDS 



1^ 
tl 



Progijml^JProgrjij 



I* 



n 



Proy m 3 Proqfm 4 



Q. 



I? 
I* 



If 



CONTAMINATION . Total negative scores 
DEFICIENCY - Total positive scores 

RATING: 




1 > Barely Important/Juat Mentioned 3 > Important/Covered In aome depth 

2 ■ Not very Important/Mentioned 4 > Critical Ne«d/MaJor focua 



EiflUrfi^-Content/Needs Comparison Worksheet 
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To perform this comparison, these steps are followed: 

1. The needs as established in the needs assessment are 
listed in the left-hand column. 

2. Each of the needs is then weighted in terms of its 
importance on a 4-point scale, with 4 being most 
important and 1 being least important. The number 0 is 
reserved for content contained in the programs, but not 
mentioned as important during the needs assessment. 

3. Each program under consideration is then reviewed to 
determine how well it addresses each of the listed 
needs. The programs are rated by a similar 4-point 
scale, with a 4 given if they cover the need in great 
depth and a 0 given if the need is not considered at all. 

4. Then content that is addressed in the programs but not 
listed on the needs assessment is added to the list. The 
weighted importance is rated as a 0, because the content 
is not important to the organization. A rating is listed 
under the program column based on how much emphasis 
is given that need or subject in that program. 

5. The negative scores for each column are summed and 
placed in the appropriate box. This represents a content 
contamination score for each program. 

6. Likewise, the positive scores in edch column are summed 
and placed in the appropriate box. This represents a 
content-deficiency score. 
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7. Next all the 1s and the 4s are reviewed. Needs identified 
as critical to the training but not met by any of the 
alternative programs should be reviewed to determine if 
any of the programs would be acceptable as is; if they 
are not acceptable as is, new programs must be 
identified or one of the programs must be customized to 
meet the critical needs. Likewise, the content- 
contamination aspects should be considered to determine 
what design changes are required in order to delete them 
from the program. 

8. Based on the scores and information concerning the above 
evaluation, the programs are given an overall rating on 
the 4 to 1 scale, and the program that provides the best 
content based on the organization's noeds receives a 4. 
This average rating could be reached by first adding up 
the values of the individual scores and then dividing that 
number by the total number of scores. 

Trainee Characteristlr s and Trainee Readiness 

The worksheet for comparing trainee characteristics and 
trainee readiness with the trainee characteristics for which the 
program is designed and the prerequisite knowledge is shown in the 
following figure. Prerequisite skills, knowledge, abilities, and 
characteristics of the trainees are listed on the sheet. Each 
program is then assessed according to the factors of trainee 
readiness and trainee characteristics. For example, the factors that 
could be considered for a class of machinists or craft workers might 
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incluc'e a high-scnool diploma, additional vocational technical 
training, an average of 3-5 years experience, no previous training in 
time management, and a general dislike of paperwork. Further, ^ 
trainees may perceive "soft" ekill training as unimportant. 



WORKSHEET TO ASSESS TRAINEE CHARACTERISTICS 
AND TRAINEE READINESS 



Program . 
Oatt 



Prtpartd by 

Ptgt cir 





PROQRAM RATING 


PftCRECHJiaiTE •KILLt/KNOWLEDOE/AIILITICt/CMAIIACTERItTaCS 


1 


1 


3 


4 










































































































































































































OVERAU 

fmno Sytiiffl: 4 « HighMt lo i • Lommi RATINQS 











Figure 7- -Tralnee Characteristics and Trainee Readiness 
Assessment Worksheet 
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Subject Matter Fypart 

Sometimes a subject matter expert may be called in to review 
the accuracy and completeness of the technical information 
contained in the programs under consideration. The following 
worksheet includes questions designed to assist this expert. After 
answG.ing the questions, this expert will be able to rate the 
programs and summarize the observations. Tr^e data on which the 
review is performed may be acquired by read-ng the materials for 
the programs, attending the programs in question, speaking with an 
instructor or designer of the program in each vendor's organization, 
or contacting previous users. 
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SUBJECT MATTER REVIEW WORKSHEET 

Program . Prepared t; 

uete Page 1 of 5 

Rate each program on a four-point ecale, l>aeed on how well 
the content of the program adherea to the following queatlona: 







Pvoorain 






1 


2 




4 


1. How aecurate and comet Is ttw InfomwHon? 
(4 • toliily accurot* to 1 ■ many InconslatanclMt) 
Coffifnont^s 










2. Hew complata la tha Information glvan? 
(4 m totally eomplaia lo 1 ■ major gapa) 
CofiHnanta* 










X Doaa tha program contain any unnacaaaary 
Information? (4 m nona to 1 « vary much) 
Cofitmanta! 










4. la lhamatarial at tha appropriate laval of 

difflailty for tha taigat population? 

(4 m appropilala lo 1 ■ way loo aaay or difficult) 

Commantsi 












Tolalafor 
pagal 











Figure 8- Subiect Matter R';view Worksheet (Page 1) 
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SUBJECT MATTER REVIEW 
WORKSHEET (continued) 



Program Prepered by 

Dote — Page _ 2 _ or _ 4 





Program 


1 


2 


3 


4 


5. How accurate ara tha axhibitt, charts, 
diagrama. and madia? (4 « totally accurata to 
1 M not accurata) Commants: 










How ralavant ara tha caaa problama to tha 
axpactad parformancaa? (4 « totally ralavant to 
1 M irraiavant) Conmanta: 










7. la tha commit tha moat up to data availabia? 
(4 m atata^of-tha-art to 1 m datad) 
Commants: 










8. Doas tha 7.t>gram contain contant that doas not 
conform to company standards and procaduras? 
(4 m nothing to 1 a a IM) Commants: 












Totals for 
psga 2: 











Figure 9--Subject Matter R wiew Worksheet (Page 2) 
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SUBJECT MATTER REVIEV/ 
WORKSHEET (continued) 

Program Prepared by 

niitA Pmqm 3 ^ of 5 






r^rogram 




1 


2 


3 


4 


9. It tht mttiritl prtstntod logically and In • way 
th«t la May to follow? (4 m y«a to 1 ■ not •% all) 
Comm«nta: 










10. Doea the program giv« trsinsaa • ch«nc« 
to teat how woll thoy havo loornod tho mattriol? 
(4 a yoa to 1 a not •% all) Commanta: 










11. Do the atudent m«tari«la saaiat th« l«aming 
•nd provide • teks-home r«f«r«nc« guide? 
(4 a •Kcellsnt mat«riela to 1 a k»ad m«t«ri«la) 
Commenta: 










12. Will • poraon eomplsting thia program b« abl« 
to perform whet h« or ah« haa l««;rned back on th« 
Job? (4 a definitely yta to 1 a d«finit«ly not) 
Commtnta: 












Totala for 
paga 3: 











Figure 10 --Subiect Matter Review Worksheet (Page 3) 
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SUBJECT MATTER REVIEW 
WORKSHEET (continued) 

Program Prepared by 

Date — Page _ 4 _ of _ 5 





Program 


1 


2 


3 


4 


13. Af all terms sxplalnsd and la a glossary 
provldad"/ (4 k yas to 1 m not at all) 










14. Is thara a bibliography and ara rafarancas 
givan whan approprlata? (4 « always to 1 « nons 
Commants: 


1 








15. Ara parformanca aids givan whara approprlata 
to raduca unnacassary study and laarning? 
(4 « axcallant aids givan to 1 « no aids givan) 
Commants: 










OVERALL RATMO lASED OH- 

4 • Aacaptabia As la 
3 « AccapUbIa with Minor Ravlslons 
2 a Accaptabia with Major Ravlslons 
1 • !«ot Accaptabia 

AOOmONAL COMMENTS GIVEN ON 
NEXT PAGE. 


Totals 

for 

psgas: 


4 










1 










2 










3 










Ovarsll 
Totals 










OVERALL 
RATING 











Figure 11 --Subject Matter Review Worksheet (Page 4) 
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SUBJECT MATTER REVIEW 
WORKSHEET (continued) 



Program 
Data 



Prepared by 

Page 5 of 



Additional comments or recommended changes or revisions: 



Program 1: 



Program 3: 



Program 2: 



Program 4: 



BftMd in part on information from Charlos F. Martlnotz't "Chocldist for Evaluation," 
Porformanoo and Instruction Journal, Juna/July I9S6, pp. 13-15. 



Figure 12 -Sub|e^t Matter Review Worksheet (Page 5) 
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Evaluation of the Criteria in the Design Dimension 
Qbjactives 

The following two worksheets are based on the discussion of 
the literature concerning objectives; they Si^e presented first to 
assess each program's stated objectives and second to summarize 
the results for multiple programs. Each objective is listed down the 
left-hand column, then the appropriate squares are checked off, 
depending on where the objective is found, what type of an objective 
it is. and whether it is stated in measurable and observable terms. 
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OBJECTIVES ANALYSIS WORKSHEET 
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TOTALS 



















Figure 1 3- -Obiectives Analysis Worksheet 
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In addition, the following worksheet allows the reviewer to 
take the information from the Objectives Analysis Worksheet and 
add it to the Objectives Comparison Summary Worksheet in order to 
make an overall rating of the programs under consideration. In 
addition to listing the number of objectives present, the reviewer 
can use this worksheet to determine what percentage of each type of 
learning domain should be in the program and then compare the 
programs on how well they meet that ratio. An example of this 
would be as follows: the relative importance of attitude toward 
customer or product or service, selling skills, and knowledge of 
product or service to a salesperson's success was determined in a 
particular situation to be 48 percent/25 percent/13 percent, 
respectively (Kerr & Burzynski, 1988). Yet the sales programs in the 
same situation emphasized skills/knowledge/attitudes at 56 
percent/43 percent/1 percent for newly hired salespersons and 65 
percent/35 percent/0 percent for salespersons with 1-3 years 
experience. None of the programs could be rated very high in 
meeting the actual need for product knowledge, but one of the 
programs reviewed might be slightly better than the others. At this 
point, the program could be given a slightly higher rating, the 
program could be customized to provide the required product 
knowledge training, or a new program could be found that better 
meets the requirements. 
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OBJECTIVES COMPARISON SUMMARY WORKSHEET 

f rogrtm f r«p«r«d by 

Dtia F«g« of 





Progrom 1 


Progrom 2 


Progrom 3 


Progrom 4 


Numbor Proton! 










Numbor Mooouroblo/ 
Oboorvoblo 










Skfii/Attltudo/Cognltlvo 
Poreontogo Comporloon 










Skill/ Attltudo/Cognltlvo 
Rotio Rating 











Ruing Syttam: 

4 • Hgh«tl to 1 • LowHt 


Progrom: 


1 


2 


3 


4 


Overall Rating: 











Figure 14- Qbiectives Analysis Summary Worksheet 

Group Presentation Methods 

The literature review yielded an extensive list of group 
presentation methods. A variety of methods will help to maintain 
trainee interest, and the methods that optimize the learning in the 
group can be selected. The leader's guide for each program can be 
scanned to consider how many methods were used in that program. 
Although the worksheet does not require that the methods be rated, 
the reviewer could rate the program in terms of what methods are 
used and how appropriate they are based on the content and learning 
objectives. 
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A program that incorporates lecture without demonstration to 
teach participants a work routine that primarily calls for using 
psychomotor skills and cognitive skills would not rate high. On the 
other hand, a program that is intended to change trainee's attitudes 
as well as to have them gain skills may incorporate many different 
methods, such as case studies, role plays, structured discussions, 
and simulations, to provide continuous stimulation and a high level 
of personal involvement. This program would rate high in terms of 
presentation methods. 

The number of different methods used in a program serves as a 
measure to indicate how well the program should be rated on this 
criterion. The appropriateness of the method to the content should 
also be considered. The following worksheet allows the reviewer to 
check off the methods used in each program. 
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GROUP PRESENTATION METHODS 



Projtct Data 

Praparad by Paga of 



Method 


Program 

1 


Program 
2 


Program 
3 


Program 
4 


L«ctur« 










fTtadlng 










Damonttration 










Raid Tripa 










Note Taking 










Opan-Foaim DIscusaiona 










Structurad Diacuaalon 










Pan^l Diacuaaion 










CaasSludtoa 










Chmtlon-Anawar Saaaiona 










Parformanca Tryout 










Brainatorming 










In-Baakata 










Incidant Procoaa 










Action Mazaa 










ToTiH Taaka/Buzz Groupa 










Aganda-a^tting Buzz Groupa 










Roltplaya 










Ravaria Rolaplaya 










Doubling Rolaplaya 










Rotation Rolaplaya 










Hot Rolaplaya 










JigaaMva 










T'Groupa 










Simulationa 










Gamaa 










Clinica 










Critical Incident 










Programmad Inatruction 










Rahbowla 










Othar 










Totdia 
OVERALL RATING 



















Figure 15 --Group Presentation Methods 
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Quality of Ingtriirtinnal Materials 

In order to review the quality of the printed instructional 
materials, the following worksheet assists the reviewer in 
considering the categories of writing style, typology, packaging, 
page organization, and use of illustrations. Each category contains a 
list of possible factors to consider (Burbank & Rett, 1986). Based on 
this information, the reviewer can summarize the information into 
an overall quality of instructional materials rating for each of the 
programs under consideration. 

A low rating on writing style may indicate materials that are 
too difficult to read, that confuse the learner, or that insult his or 
her intelligence. A low rating on typology may indicate material 
that is fatiguing to the eyes. The wrong type size or the wrong style 
can interfere with the trainee's deciphering the message. A low 
rating on packaging may suggest that materials that are important 
for on-the-job applications may not hold up under such conditions. A 
low rating on page organization may indicate that materials lack 
continuity of design, such as headings that are not of the same type 
size in different chapters. A low rating on the use of illustrations 
may indicate materials that do not focus on the critical elements of 
the instruction in a manner that is readily understandable. The 
ratings from each of the above categories are then combined by the 
reviewer to reach an overall rating for each program. 
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QUALITY OF INSTRUCTIONAL 
MATERIALS WORKSHEET 



Project Date 

Prepared by Poge I of 2 

Rating Syti^: 4 ■ high Iq 1 ■ low 

WrHIng Style: Program: 

12 3 4 

Adapts to I— mor>' r—ding laval 

Uaaa short, oof^ta ttatamante 

Uaaa ooncmta, tamiiar languaga ■ 

Dafinaa naw torma 

Utaa axamplaa whan poaaibla 
Uaaa oonaiatont wonfing 

Uaaa acMva languaga 

Employa *you" appropriately 

DMdaa long Hata Into groiipa 

Uaaadtograma 

Uaaa varbal analogiaa 

Total I 



Typology Program: 

12 3 4 

Typa siza and spacing 

Typafaoa and styla 

Una and paga langtft 

Contrast 

Total 



Packaging Program: 

12 3 4 



Siza 










Covar stying/attractivanass 










Covar matarial 










Quafity of papar 










BkvJng 










Total 











(continued) 



Fiyura 16 --Quality of Instructional Materials Worksheet 
(Page 1) 
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QUALITY OF INSTRUCTIONAL 
MATERIALS WORKSHEET (continued) 

Project 



Prepared by 



Page Organization 



Data 
Page 2. 



of. 



2_ 





4 

I 


z 


9 


M 

4 


L^yvui wiTiiiv II VIII psQV HI psQV 










Logical Ofdtr apparent to lMm»r 










Numbtrt. haadings. buNatt, ale. utad 










Diffarant kJaai daariv taociratad 










Tabla of Coolants uaad 










Indaxuaad 










WNia spaoa uaad affactivaly 










Non-taxt atamantt ralatad to tax! 










Pagas visually balancad 










Simplldty of ovaral datign UMd 










Rafarancat/BibKography usad 










Total 










llluatratlona 


1 


program; 

2 3 


4 



Raiavant to oontant 










Rt laamars* capabHitias 










Easily undaralood 










Focus laamar*s attantion quickly 










Ralals parts logically 










Appropriata laval of oomplaxlty usad 










Laamar Idanlillabla cuas usad 










Color usad 










Total 








OVERALL RATING SUIMiMARY 


1 


Program: 
2 3 


4 



WHUngStyla 



Typology 



Packaging 



Paga Organi2atk>n 



Hhistrattons 



Ovaralt Rating 



Figure 17->Quality of Instructional Materials Worksheet 
(Page 2) 
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structuring tho r.nn<pnt 

A review of each program's structure begins by creating an 
outline. Often there will not be an outline in the program materials 
and it must be c<'eated by the reviev^rer. An overview of the main 
elements that form the model on which the program is based can be 
considered, and they can be diagrammed or listed on the following 
worksheet. Examples of questions to consider in rating each 
program on this factor include: (a) Does the structure allow the 
trainee to relate the content to previous knowledge? (b) Does 
content structure minimize confusion for the trainees? and (c) Does 
the content structure make it easy for the trainee to conceptualize 
the training, thus increasing the likelihood that he or she will 
I'emomber and apply the training on the job? The instructional 
sequencing factor considered on the next worksheet addresses the 
nuts and bolts of how the training program is put together. In it, 
emphasis with structure is on the larger picture, the basis of the 
program's conceptualization that gives direction to the 
implementation strategy. 
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CONTENT STRUCTURING WORKSHEET 

Program Prepared by 

Dote^ Page of 



Create a diagram or outline of how each program is structured. 
Program 1; program 2: 



Pmrm 3; Program 4; 





Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











£iauia_Lfl--Content Structuring Worksheet 
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Instructional SftgnflncinQ 

In addition to considering the structure on which the program 
is modeled, an evaluation of how the program is sequenced can be 
made using the following worksheet. A set of sample sequencing 
strategies is listed at the top. By reviewing the program materials 
and outline, a diagram or list can be made of how and what type of 
sequencing strategy has been used to implement the program. 
Questions to consider in arriving at the overall rating for each 
program include: (a) How easy will it be for the trainees to follow 
the program conceptually? (b) How will the trainees react to the 
sequencing strategy experientially? and (c) Is the sequencing 
strategy appropriate for the content? 

As with content structure, tho comparison of several different 
programs will heighten the reviewer's understanding and ability to 
rate the differences. Just as American managers gained 
considerable insight into their profession by comparing themselves 
with Japanese managers, comparing programs will result in an 
increased understanding of each aspect of the programs. In this 
case, the first sequencing strategy considered may appear good 
enough to the reviewer, but when compared to a different strategy, 
the reviewer may gain new insight into why one strategy may be 
more appropriate than another. 
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INSTRUCTIONAL SEQUENCING 
WORKSHEET 



Program 
Dote 



Prepared by 

Page of 



How are the elements of the program sequenced? 



" Logically 

• Step-wise progression 

• Whole-part-whole 



Program 1: 



• Key questions 

• Inductive (experiential) 

• Problem-oriented 



Program 2: 



Program a- 



Program 4: 





Prog 


ram: 




1 


2 


3 


4 


Overall Rating: 











£iflll£fi_ia" Instructional Sequencing Worksheet 
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Rate of Delivflry 

Rate of delivery concerns both the speed at which the program 
is delivered and the size of each instructional segment. How fast a 
program should be delivered depends on several variables, such as 
trainee characteristics factors, trainee readiness factors, 
importance of the instruction, and complexity of the material 
presented. Instruction that is delivered too fast or too slow may 
frustrate the trainee, resulting in less ei'ective training. 
Instruction that is sized into very large segments may confuse 
trainees unnecessarily or require a concentration level beyond the 
trainees' capabilities. The following worksheet allows the reviewer 
to consider each element in arriving at an overall rating for each of 
the programs being considered. 
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RATE OF DELIVERY WORKSHEET 



Program 
Date 



Prepared by 
Page 



of 



What It tha expected rate of learning? 

How appropriate It the rate of learning to the content? 

Proqrem 1: 
Program 2: 
Program 3: 
Program 4: 

How largo It aach Inttructlonal "placa"? 

It each Inttructlonal "place" appropriately tized? 

Program 1: 

Program 2: 

Program 3: 

Program 4: 



Prog 



ram: 



1 



2 



3 



4 



Ovarall Rating: 



Figure 2Q--Ratft of Delivery Worksheet 
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Repatition anri Prafttiftfl 

The purpose of practice is to reinforce the training. By 
providing actual experiences with the program, trainees are more 
likely to apply the skills or knowledge successfully. The purpose of 
practice and repetition in training is to translate the knowledge 
gained into effective action. In highly effective training, trainees 
are not just passive elements; instead, they are integral parts of the 
program. In effective training, trainees are actively involved in the 
learning process. 

The repetition and practice worksheet allows the reviewer to 
consider how practice will be conducted in various programs. To 
what degree does the program allow the trainees to experience and 
practice the learning? The following worksheet compares programs 
on the basis of what will comprise the practice, how the practice 
compares to the actual experience, and how the trainee is given 
feedback on his or her performance. This information is then 
combined into an overall repetition and practice rating for each 
program. A high rating on this factor increases the likelihood that 
the trainees will use the training on the job. 



138 



REPETITION AND PRACTICE 
WORKSHEET 



Program 
Date 



Preparer* by 
Page 



of 



What will tha practico conalat of? 
la It an appropriata amount, too littia, or too much? 
Program 1: 

Proa""' 2: 
Program 3: 
Proanm 4: 

In what waya will tha practice performance differ from the 

actual Job performance? Are the differencea acceptable 

to the reviewer? Do they need to be or can they be improved? 

Program 1: 

Program 2: 

Proanm 3: 

Program 4: 

What kind of and how often will feedbacic be given to the 
trainee during the practice? la thia achedule acceptable 
or can it be improved? 
Program 1: 

Program 2: 

Program 3; 
Program 4; 





Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











Figure 21 -Repetition and Practice Worksheet 
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Reinforcemflnt and Rftwards 

In addition to how the skills and knowledge gained in the 
training are practiced, attention may be given to how the trainee is 
motivated, reinforced, and rewarded. The following worksheet 
allows the reviewer to evaluate each program against a set of six 
motivational elements. The elements are divided up according to 
whether they are critical at the beginning, middle, or end of the 
training (Wlodkowski, 1985b). 

In the beginning stage, trainees' attitudes and the trainees' 
needs and desires must be addressed in order for the participants to 
buy into the learning process. Providing an environment in which 
people feel safe and the risks of the training are minimized will 
assist in creating favorable attitudes. Justifying the training by 
building on the trainees' existing needs and desires will do much to 
ensure their interest and attention throughout the program. 

In the middle stage of the program, reinforcement should focus 
on the steps provided to maintain the trainees' attention. They may 
be stimulated by using a variety of learning techniques and methods. 
At the same time, the trainees' emotional reaction to the training 
should be considered. Oftentimes, affective aspects of the training 
programs are not considered. It may be difficult to measure this 
aspect of the program because measuring it may depend more on the 
instructor than on the materials. Documentation to assess this 
aspect of the program would most likely be found in the comments 
accompanying the leader's guide. If such aspects were not 
considered in the program's material, that program would likely 
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receive a lower rating. Another measure of the affective learning 
factor may be taken from the evaluation ratings of trainee's 
reactions to the program. 

During the end stage of the program, reinforcement and 
rewards should be directed toward encouraging and supporting 
competent learning. As with the repetition and practice v/orksheet, 
this aspect focuses on how the successful behavior and learning are 
given positive reinforcement. How is unsuccessful behavior 
addressed? 

Based on a review of these six elements of a program's 
motivational strategy, an overall rating of each program can be 
given. A program that includes a clearly defined motivational 
strategy during the beginning, middle, and end stages is more likely 
to be effective than a program in which the strategy is left to 
chance. 



o 
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REINFORCEMENT & REWARDS 
WORKSHEET 

Program Prepared by 

Date Page of 



Rate each program according to how well each motivation factor 
ia incorporated into the inatructional deaign. 



B 

E 1. AtliludMaradMit with by minimizing 

0 ttw risk of tho training and providing a Mfo 

1 •nvirurmant for tralnata to laam. 
N 

^ 2. The training It doalgnad torn what 
' pacplawanttDdo. M builds on tralnaaa' 

Z nr^s and daslrss. 

0 


Progrsm 


1 


2 


3 


4 










9. Tha program Is daslgnad to males thaprocass 

of laaming sttmulaling and axcllf ng and lo 

' maintain atlsntlon. 
D 

D 

t 4. Tha amotfonalaapaets of ths laaming art 
C eonsldarad by balancing diacovary and 
analytical aiparlancas. 










5. Tha objacttvas ara claarty stalsd and 
oommunicatad so lhat Ilia tralnaaa will 
^ aiparlancacompalantpartormancaof tha 
1 aMis or laaming covarad In tha program. 

N 

^ a. Daslrad bahavlors in tha training ara 
poslttvsly ralnforcad by tha Instnictor. 











Rating System: 
4«Hlqhsstto1 -Lowast 


Program: 


1 


2 


3 


4 


Overall Rating: 











! Figure 22- Reinforcement and Rewards Worksheet 

I 
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Adult Learnirig Prinriples 

Based on the summary of adult learning principles provided in 
the literature review, the following worksheet allows the reviewer 
to consider each program on the basis of each of the identified 
principles. The first question considers the climate established in 
the program. A climate of respect exists when the trainees are 
treated appropriately. Appropriate treatment does not include 
behaviors that belittle others or involve emotional or physical abuse 
in any way (Brookfield, 1986). The second question addresses the 
use of a collaborative mode of learning, which exists when the 
trainees and facilitators are "engaged in a cooperative enterprise In 
which, at different times, leadership and facilitation roles will be 
assumed by different group members" (Brookfield, 1986, pp. 9-10). 
Question three is concerned with developing a sense of 
empowerment and self-directedness in the trainee. Trainees are 
encouraged to take responsibility for their own learning, ensuring 
that the training meets their needs. Rather than seeing themselves 
as reacting to the "uncontrollable forces of circumstances," they 
should see themselves as "proactive, initiating individuals engaged 
in a continuous re-creation of their personal relationships, work 
worlds, and social circumstances" (Brookfield, 1986, pp. 9-11). 
Question four is concerned with how well the learning is integrated 
with the trainees' past experiences. Training that builds on 
trainees' previous experiences is more likely to be accepted and 
found meaningful than passively acquired learning. Question five 
considers the degree to which the trainees participate in the 
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program. Participation can be increased by encouraging trainees id 
developing "their own learning objectives, planning their own 
learning programs and evaluating their ^progress" (Mezirow, 1981. pp. 
21-22). Question six addresses the program's emphasis on critically 
reflective thinking and the "relevant social and cultural norms 
(contextual factors)" (Yeres, 1986, p. 16) that may affect the 
learning. Brookfield (1986) defined this principle as follows: 
"Through educational encounters, learners come to appreciate that 
values, beliefs, behaviors, and ideologies are culturally transmitted 
and that they are provisional and relative" (pp. 9-10). Question 
seven is concerned with the training being action oriented. To what 
degree does the training focus on action ou'tcomes, such as better 
decision making, on making behavioral changes, or on increasing 
trainees' abilities to learn? Finally, question eight is concerned 
with how true to life are the problem-posing and problem-solving 
aspects of the program. By making the problems as real as possible, 
the likelihood of on-the-job application of the learning is increased. 
After each program has been considered for each of the questions, 
the results are combined into an overall rating. 
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ADULT LEARNING 
PRINCIPLES WORKSHEET 



Program Prepared by 

Date Page of 



Rate each program on how well each principle of adult 
learning la uaed in the program. 





Program 


1 


2 


3 


4 


1. A cllmato of rMp«ct It malntalntd. 










2. A eoll«bor*uir« mod« of iMmIng It uttd. 










3. Mf-directtdnttt and tht tmpowtrmtnt of 
tht Itamtr It fotttrtd. 










4. Tht tralntet* patt tiparltfictt art uttd at 
a dtparturt for tht Itarning tiptrltnctt. 










S. Particlpatiofi It uttd at much at pottlblt. 










S. Thtrt it an tmphctit on rtfltctlvt thinlcing 
and an awartnttt (J conttitual fictort. 










7. A conctm for Itaming with action (Including 
dtclt^on making, bthavloral changt, 
ability to Itam, and colltctlvt action). 










t. Rtal probitmt art uttd at tht batit of 

probltm potlnjB and probltm tolvlng. 










TOTALS 











Rating Systtm: 

4 - Exotlltnt to 1 - Poor 


Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











Figure 23- -Adi]it Learning Principles Worksheti 
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Media Usq 

An evaluation of the media used in the program may be 
performed using the following worksheet. The simplest way to use 
this sheet is to list each type of media used in the program. If other 
considerations are important, such as cost, in-house media 
capability, technology, flexibility, and so forth, it may be necessary 
to weight appropriately the various media choices. This could be 
done by assigning a weight factor on the line next to each type of 
media, according to the specific situational requirements. Then the 
rating factor could be multiplied by the rating within each box and 
the weighted totals could be together at the bottom. If the In-house 
capability to utilize the media provided in the program is not 
available, (i.e., the company has no videodisk player), the program 
could be eliminated from consideration, or at least a cost estimate 
could be made for rectifying the situation. 
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MEDIA USE WORKSHEET 

Program Prepared by 

Date^ Page of 



Typ« off M«dia 


Program 

1 


Dr A Ar A fVl 

2 


Program 

3 


Progr a m 
4 


Ovoftmd Trinttpmnci«8 










Op«qu« and RMr>Scm«n Projectors 










Chalkboards Mnd Olhsr Doards 










Fiipcharts 










Slidas 










Flimstrips 










Motion-Plctura Films 










VUsoiapas 










Intaractiva Vidaodisks 










Audtolapaa 










Printad Matarials 










Modals. Simulators, and Raal Equipmant 










Computar Softvvara 
























































































































































Totsit: 










Rating: 











Rating System: 4 - HIghatt to 1 - Lowast 



Figure 24 -Media Use Worksheet 
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Evaluation of the Criteria in the Context Dimension 
The context dimension considers organizational factors 
associated with the buying/decision-making process. What factors 
in the buyer's organization, what factors in the vendor's 
organization, and what factors in the relationship between the two 
organizations will affect the training program's overall 
effectiveness? For example, in an attempt to improve customer 
service quickly, it is not uncommon for a company to offer customer 
service trainirig vor their front line employees. Yet, an analysis of 
the situation may show that the people on the front line already 
possess these skills, and it is the "system" that does not allow them 
to use the skills. The problem may stem from the support they 
receive from the company, the quality of the products, or the 
management style of the company. For example, an airline decided 
to improve customer service by initiating a companywide service 
training effort for front line employees. The airline's plan did not 
address the more serious questions, for example, flights that did not 
arrive on time and baggage that was lost. This company needed a 
clear assessment of the organizational factors associated with the 
training. 

In addition to the organizational factors, the context 
dimension considers the implementation factors that are critical to 
a program's success. Even the best program will fail to achieve the 
full potential of its effectiveness if it is poorly implemented. 
Especially in human services and social programs, implementation is 
a major factor in program effectiveness. "Evaluations that have 
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ignored implementation issues may have asked the wrong questions" 
(Patton, 1986, p. 148). Factors from the literature review that will 
be considered in this dimension include: price, vendor's ability to 
deliver, cultural fit, instructor training, vendor's capability, and the 
vendor's ability to customize the program. 
Price Comparison 

The price comparison worksheet is a tool for comparing the 
costs of programs. However, many factors can affect pricing, and 
the price of a program for a particular organization may be radically 
different from the price of the same program for a public seminar. 
If known, the more accurate price (the actual cost for the company) 
should be used. If the programs differ in length, based on differing 
amounts of instruction, the price could be pro-rated in order to find 
a standard figure, such as the price for eight hours of instruction; 
the comparison can then be based on equal measures. The critical 
Issue is the total price required to acMeve the program's objectives, 
and it is important to determine what is included and excluded in the 
price. If in-house staff are trained by the vendor to deliver the 
program, there may be asi?ociated costs for the length and quality of 
Instructor training, leader's guides, and administration kits for each 
instructor. For programs that continue over a period of time, the 
costs of the participant's materials may be the critical factor In 
pricing. Often, a program will be customized, so the cost of 
customization represents the major portion of the start-up costs, 
and the cost of materials will be minimal. Each situation will have 
Its own unique aspects. After considering the price in relation to 
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the quality of each item for a specific situation, a rating is assigned 
to each variable. An overall rating is then assigned. 

A word of caution is noted regarding the significance of the 
pricing worksheet: After one organization evaluated the design and 
content of a specific program, the vendor instantly dropped the 
price. In addition, the vendor, interested in the report, was willing 
to pay for a copy of the evaluation. The knowledge gained by 
performing the type of evaluation suggested in this study may put 
the decision maker in a more powerful position to negotiate price 
with the vendor. 
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PRICE ANALYSIS WORKSHEET 



Program Prepared by 

09\B Page of 





Program 1 


Proa ram 2 


Praarsm ^ 




Seminar 
(Inst. + Maris) 




$ 




s 


Pro-rated 

fnr ft Ure 

■or o nrs> 




$ 




$ 


Instructor 

1 1 a III 1 iiy 




$ 




$ 


Administration 
Kit 




$ 




$ 


Materials 
Only 




$ 




$ 


Other (I.e., 
Customizing) 


$ 


$ 




$ 


OVERALL 
RATING 











Rating System: 4 . Highest to 1 . Lowest 



Figure 25-Prinft Analysis Worl^sheet 
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Ahility to Deliver 

The time in which a program can be most effectively 
implemented may be an important factor in the decision-making 
process. Once a training need is identified, the internal costs of 
delivering a product or service may be viewed as excessive. Thus, 
excess costs that are accrued during the time that the need is not 
met should be considered, along with the costs associated with the 
actual purchase and implementation of the program. The time to 
implement may depend on each vendor's purchasing procedures and 
capabilities. If the program is customized, the time required for 
that process should be considered. 

If the training is provided by a vendor, when are their 
instructors available? If the Internal staff will be trained, when 
and where will the training take place? When is management 
willing to release the employees for this training? After 
considering the relevant factors, a rating is assigned each program. 
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ABILITY TO DELIVER WORKSHEET 

Timeframe desired: Latest acceptable date: 



Program Prepored by 

Date^ Page of 





Program 1 


Program 2 


Program 3 


Program 4 


Seminar 
(Inst. -I- Mat'ls) 










Instructor 
Training 










Materials 
Only 










Customizing 
Materials 










Producing 
Video 










Otiier 











Rating System: 

4 m Highest to 1 « Lowest 


Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











Fi9urQ 2fi--Ahiiity to Deliver Worksheet 
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riiltnral Risk Assessment 

Programs that are most important to an organization and have 
the highest risk of failure are often those that are least compatible 
with the existing culture. The decision maker should reconsider 
purchasing such a program and make an effort to adapt the program 
more closely to the existing culture. If decision makers understand 
their cultures, the following worksheet could help them to rate how 
the programs under consideration would fit their organization. 

The format of this worksheet is a matrix that considers the 
risk associated with two factors: compatibility of the program with 
the culture and the importance of the training. A program that is 
low in compatibility with the culture and high in terms of 
Importance to corporate strategy would represent an unacceptable 
risk, and the program should either be removed from consideration 
or redesigned to be more compatible with the existing culture. 
Programs that are somewhat compatible with tne culture and are of 
medium importance are manageable risks. In general, programs 
rated as a manageable risk offer the highest potential for being 
effective: It is more likely that the results of such programs will 
be used and accepted in a way that shifts the organization's behavior 
In the desired direction. Programs that represent a high 
compatibility with culture and are of low importance to the 
organization are rated with a negligible risk, in such cases, the 
reviewer may suggest programs that are more important to the 
organization. 
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The matrix also shows that programs of high importance 
should be designed to be highly compatible with the existing culture 
in order to decrease the risk that the training effc* will fail or be 
ineffective. The decision maker should seriously reconsider 
programs that are both low in importance and not compatiblti with 
the existing culture: Even if the risk is manageable, there may be 
more fertile areas for training resources. 

The boundaries between negligible, manageable, and 
unacceptable risk depend on many factors within both the 
organization and the programs under consideration. In addition, risk 
may be high for the training department, but low for the 
organization, or vice versa. It is important that the decision maker 
consider a risk factor in relationship to the program selected: What 
are the risks, and whom do they affect? If properly identified, 
aspects of the training program or the implementation of the 
program may be reconsidered to decrease the risk (Davis, 1984). 



155 



CULTURAL RISK ASSESSMENT 
WORKSHEET 

Program Prepared by 

Date Page of 




High Medium Low 



Compatability with Culture 

Plot each of the programs under consideration on the above 
matrix. Steps that could be taken to decrease risk include: 

1. 
2. 
3. 



Rating System: 

4 > Highest to 1 > Lowest 


Program: 


1 


2 


3 


4 


Overall Rating: 











Figure 27 --Cultural Risk Assessment Worksheet 
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Instructor Training 

it may be important fc* the supplier to teach in-house people 
to provide the training. The following worksheet contains questions 
to help the reviewer determine the role of the instructor and 
aspects of the instructor training, if the vendor provides it. How 
will feedback be given? What are the prerequisites? What are the 
qualifications of the person who will conduct the instructor 
training? How much time will be spent on the training? When and 
where will it be held? Each question should be considered from two 
perspectives: Quality and cost. For example, it may be much more 
cost effective to have the instructor training provided in-house than 
to pay for instructors' expenses at an alternate location. 
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INSTRUCTOR? TRAINING WORKSHEET 



Project , 
Dote 



Prepared by 



Page I of 2 



ho Instructors rolo? 


Program 


1 


2 


3 


4 


Consultsnt 










Contont Export 










Lecturer 










Evs^'iitor 










Facllitstor 










Coach 










Entortainer 










Othor 































W tha wndor provide Inatrueter trainin g, tha follnwlnq 



auMtlona mav t» con«ld«red: 



How will fMd^ck be given to the Instructors during the training? 
Is this accep able? Can it be Improved? 



How Is fMdback glvMi? 



Program 1: 



Program 2: 



How can It to Improvod? 



Program 3: 



Program 4: 



Figure 2S- -lMstructQr Training Worksheet (Page 1) 
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INSTRUCTOR TRAINING WORKSHEET 

(continued) 



Project, 
Date 



Prepared by _ 
Page 2 



of 



What are the prerequisites for Instructors? 
Program 1 : 
Program 2: 
Program 3: 
Program 4: 

What are the v ueiification i of the trainers of the new instructors? 

Program 1: 
Pfooram 2: 
Program 3; 
Program 4: 



Program 1: 



£rQflcanL2: 



Program ' 



Proaranr 4; 



Wiwnwiii th 
training b* Md-/ 



How lonfl will 
th* training last? 



Wlwrawlll tti* 
training b« hald? 



Ftating System: 

« « Highest to 1 -io.vest 


Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











Figure 29- - Instructor Training Worl<sheet (Page 2) 
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Vftndor's Capability 

An evaluation of each supplier's resources, personnel, and 
technological capabilities can be performed using the following 
worksheet. Notes can be made about why specific ratings were 
given. For example, the supplier for Program 1 may lack the ability 
to provide interactive video, which results in a lower rating. If it 
were critical that the supplier have this capability. Supplier #1 
would be eliminated from further consideration. The information is 
then summarized into an overall organizational capability rating for 
each supplier. 
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VENDOR CAPABILITY WORKSHEET 



Program 
Date 



Prepared by 
Page 



of 



Rata each program In tarma of tha aupp^lar'a raaourcaa: 
Program 1: 

Program 2: 

Program 3: 

Program 4: 

Rata each program In tarma of tha auppliar'a parsonnei; 





1: 


Program 


2: 


Proaram 


3: 


Proaram 4: 



Rat* Mch program In terms o? tha supplier's technology to do the Job? 
Pre .am 1: 

Program 2: 

Program 3: 

Program 4: 



Rating System: 

4 - Hichest to 1 - Lowest 



Program: 



1 



2 



3 



4 



Overall Rating: 



Figure 3Q--Vendnr Capability Worksheet 




Customizing Capability 

The following worksheet is designed to assess an overall 
customizing capability rating for each program. If a program must 
be customized to meet the identified training needs, an assessment 
of all the programs under consideration should be made. The 
customization rating scale (1 to 6 at the top of the worksheet) 
refers to various levels of change. Although vendors may state that 
their programs are easily customized, the buyer should determine 
the true degree of work and expertise required to customize a 
program. Each number of the customization rating scale represents 
an increased degree of effort in the custom designing process. A 
major change in the length of the program would require more effort 
than simply adding in cases specific to the purchasing organization. 
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To use this worksheet, first list the numbers associated with 
the custom work required. For example, if a program needs to be 
shortened, specific cases changed, and the company name changed, 
but no additional video needs to be shot and no new content created, 
the rating would be 1, 2, 5. The cost column is based on the vendor's 
estimate to perform the customization The costs may be stated in 
torms of performing the custom work in-house, having the vendor 
perform the work, or as a ratio by putting a slash in the box and 
listing the estimated vendor cost at top and estimated in-house cost 
below. In certain situations, when the work can be delivered may be 
more important or as important as the actual dollar costs. The 
column headed Delivery Date is used to consider this aspect. Based 
on these aspects of the decision, an overall rating is then assigned 
to each program for the customizing capability rating. 
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CUSTOMIZING CAPABILITY WORKSHEET 

Program Prepared by 

F^ate Page of 

6 



0 


1 


2 


3 


4 


5 


Nothing 


Change 


Add 


Add 


Add in 


Major 


can be 


Company 


apeclfic 


specific 


new 


change in 


euttoiPtaed 


Name 


casea 


video 


content 


length, video, 



and/or content 



Total 
custom 
design 



Index 
Rating 



Program 1 



Cost 



Delivery 
Date 



Other 



OVERALL 
RATING 



Program 2 



Programs 



Program 4 



Rating System: 4 » Highest to 1 > Lowest 



Figure ai- Customizing Capability Rating Worksheet 
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Evaluation of the Criteria in the Control Dimension 
The control dimension concerns the evaluation and research 
that have been conducted during a program's development. In 
addition, aspects of evaluation and design that pertain directly to 
the program's effectiveness are considered. According to the 
literature search, the factors indicated within this dimension 
include needs assessment/research, effectiveness 
evaluation/research, checking references of previous users, transfer 
of learning, and testing and measurement methods used in the 
program. 

Needs Assessment/Research 

Before a program is developed and designed, certain 
assumptions are made about -he needs that the program will 
address. The supplier of a program should have conducted a needs 
assessment as part of the development process. Reviewing the 
documentation for the needs assessment can provide some insight 
into the rationale for the program and how closely the needs 
assessment from which the program was initially designed 
compares with the buyer's needs assessment. In the following 
worksheet the reviewer evaluates the needs assessment regarding 
the clarity of intent, statement of purpose, sample size and 
characteristics, methodology, interpretation of the data, and results 
and conclusions. 
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NEEDS ASSESSMENT REVIEW 
WORKSHEET 



Program 
Date 



Prepared by 
Page 



I. INTRODUCTION: 
Does the introduction to the needs assessment: 

a. Review the relevant literature? 

b. State the preliminary questions? 

c. Justify the problem as important and worthwhile? 

d. Relate the problem to theory? 

e. Use Inclusive language? 



II. SAMPLE: 

a. Is the assessment sample large enough? 

b. How was the sample population for the assessment chosen? 

c. How does the sample population compare with the 
intended audience within the purchasing organization? 

d. Is there evidence of sample biases? 



III. DATA COLLECTION METHODS: 

a. What methods are used to collect data? 

b. Is there evidence that the methods are valid and reliable? 

c. Are there specific wealcnesses inherent in the methods? 

d. If there are weaknesses, have they been addressed? 



Figure 32 -Needs Assessment Review Worksheet (Page1) 
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NEEDS ASSESSMENT REVIEW 
WORKSHEET (continued) 

Program Prepared by 

Dete Page of 

IV. PROCEDURES FOR INTERPRETING DATA: 

a. What procedurea were uaed to interpret the data? 

b. la there evidence that the procedurea are valid and reliable? 

c. Are there weakneaaea in the interpretation that may biaa 
the reaulta? 



V. RESULTS AND CONCLUSIONS: 

a. Are the reaulta atated clearly and in understandable 
terma? 

b. Doea the inveatigator exaggerate, conclude too much, 
cr otherwiae add bias to the needa assessment 
findinga? 



VI: OVERVIEW: 

a. What are the main strengtha of the needs asseaament? 

b. What are the moat aerioua deficienciea? 

c. How do the deficienciea, if any, affect the application of 
the needa asseaament to the purchaser*a situation? 
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NEEDS ASSESSMENT SUMMARY 
WORKSHEET 



Project 
Dote — 



Propored by 

Poge of 



Summary comments and notes: 
Program 1: 



Program 3: 



Program 2: 



Program 4: 





Program: 




1 


2 


3 


4 


Overall Rating: 











Figure 34 --Needs /^s'iessment Review Summary Workslieet 
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Evaluation/Rasearch 

In addition to a needs assessment, an existing program may 
have been previously evaluated for the results produced by the 
training. Indeed, one advantage of using a vendor-supplied training 
program is that it has been used in other organizations and that 
improvements probably have been made based on how the program 
worked in these previous situations. If such a previous evaluation 
research was done, the decision maker should .isk to see the results. 
If the vendor is not able to provide such supporting evidence, the 
reviewer should consider carefully any claims made about the 
results of the program. If a previous evaluation was made and the 
vendor is willing to share the results with the customer, the 
following worksheet provides a means for reviewing it. A series of 
questions assesses the introduction, the sample populations, the 
measures, the treatments, and the results and conclusions. On the 
final page, comments and an overall rating for each program are 
made. 
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EVALUATION RESEARCH REVIEW 
WORKSHEET 



I. INTRODUCTION: 
DoM the Introduction to the research 
report: 

a. Review the relevant literature? 

b. State the problem clearly? 

c. Justify the research as Important and worthwhile? 

d. Rotate the problem to theory? 

e. Control for bias In the study by using unbiased language? 



II. SAMPLE: 

a. Is the sample large erK)ugh? 

b. How was the sample population chosen? 

c. How does the sample population compare with the 
intended audience within the purchasing organization? 

d. Is there evidence of sample biases? 



III. {MEASURES: 

a. Is there evidence that the measures are valid and reliable? 

b. Are there any specific weaknesses In the measures? 



Figure 35 -Evaluation Research Review Worksheet (Page1) 



Program 
Date 



Prepared by 
Page 



of 
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EVALUATION RESEARCH REVIEW 
WORKSHEET (continued) 



IV. TREATMENTS: 

a. What type of rasearch design was used? 

b. Are the treatments described In sufficient detail? 

c. Are there weaknesses In the evaluation design that may bias 
the results? 



V. RESULTS AND CONCLUSIONS: 

a. Were appropriate statistical tools used? 

b. Are the results stated clearly and In understandable 
terms? 

c. Does the Investigator exaggerate or concluoe too much 
from the findings? 



VI: OVERVIEW: 

a. What are the main strengths of the research? 

b. What are the most serious deficiencies? 

c. How do the deficiencies, If any, affect the application of 
the research to the purchaser's situation? 



Figure 36-EvaluatiQn Research Review Worksheet (Page 2) 



Program 
Date 



Prepared by 
Page 
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EVALUATION RESEAHCH REVIEW 
SUMMARY WORKSHEET 



Projtct Prepared by 

Otite Page 3 of 3 

Summary comments and notes: 



Proyram 1: 


Program 2: 






Program 3: 


Proaram 4: 







I 





Program: 




1 


2 


3 


4 


Overall Rating: 











Figure 37-Evaluation Research Review Summary Worksheet 
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References 

One of the advantages of buying a program from a supplier is 
that the program has been used by other organizations. The next 
worksheet provides a list of possible questions for other users of a 
program. After contacting a reference, the decision maker should 
make clear who he/sli^ is, why he/she is calling, and how the 
information will be u.-sed. The purpose of the ref(»rence checking 
worksheet is to assist the reviewer in asking detailed questions. It 
is better to ask directly, "What was management's reaction to the 
workshop?" than to ask, "How did you like it?" After checking with 
other references, the reviewer summarizes the information into an 
overall rating. 
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REFERENCE CHECI 
Contact 

Program Pi 
Contact #1: 

NAME 
POSITION 
COMPANY 
ADDRESS 

TELEPHONE 
DATE CONTACTED 
NOTES 


(INQ WORKSHEET: 
Listing 

rnparRd hy 

•ag Of 

Contact «2: 

NAME 

POSITION 

COMPANY 

ADDRESS 

TELEPHONE 
DATE CONTACTED 
NOTES 


Contact #3: 

NAME 

POSITION 

COMPANY 

ADDRESS 

telephone 
;ate contacted 

NOTES 


Contact «4: 

NAME 

POSITION 

COMPANY 

ADDRESS 

TELEPHONE 
DATE CONTACTED 
NOTES 



Figure 38 --Reforence Checking Worksheet: Contact Listing 
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REFERENCE CHECKING WORKSHEET: 
Interview uuide 



Reference Prepared by 

Dete^ Page of 

I.What were the participants' reactions to the training? 



2. What was management's reaction to the training? 



3. Old the training result In any change In work behavior? 



4. How did the program meet your expectations basod on the vendor's 
assurances? 



5. What benefits do you fbel your organization gained by using 
this particular program and/or vendor? 



6. What was this program's biggest strength? Biggest weakness? 



7. Would you use this vendor again for other training programs? 
Strong Yes Yes Maybe No Svrong No 

8. Based on your Investment of time and money, how would you 
rate this program overall? 

Superb Very Good Good Fair Poor 

9. Is there anything else I should know about this program and vendor? 
(Additional comments can be written on the back.) 



Figure 39-Reference Checking Worksheet: Interview Guide 
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REFERENCE CHECKING WORKSHEET: 

Summary 

Reference Prepared by 

Date Page of 





Program: 




1 


2 


3 


4 


Overall Rating: 











Program 1: Program 2: 



Program 3: Program 4; 
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Fiqurfe 4 0--Reference Checking Worksheet: Summary 
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Transfer of Learning 

The critical concern of a training program is whether the 
training is actually applied and used on the job. This next criterion 
indicates how well the program facilitates transfer of learning to 
the joD. Of the several models of assessing transfer of learning that 
were presented in the literature review, two methods are considered 
in this section: using field analysis and using factors that affect 
performance. 

If Lewin's field analysis model is used when the reviewer 
considers the transfer of learning from a training program, the 
issues on the following worksheet are raised (Sredl & Rothwell, 
1987). The steps described below would be helpful for the reviewer: 

1. Answer each of the four questions concerning internal, 
induced, restraining, and driving forces. 

2. Rate each program according to the four-point scale = 
excellent and 0 = doesn't address the issue) based on how 
well the program addresses that particular force. 

3. Provide an overall rating of the programs from best to 
worst (4 to 1). 
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TRANSFER OF L PARNING- 








FIaIH Andlvfiifi WorkshAAt 








Environmtnt^V 








/ /IndlviduaiX \ 








Driving Forces J / _ A ^ 1 ^ Restrainino Forces 




I Y Internal /nduoed/ 








\ \^ Forcer/ forcey 








Figure. Lewin's Model of Change 




RATI 


NGS: 


















I.What forces are acting to favor on-the-job appli- 


E 
m 


E 
• 


E 
• 


E 
S 




cation of the learning acquired in this training? 


w 


Q 


Q 


Q 

0 




(DRIVING FACTORS) 


s 

^ — 


s 

a. 


g 


a 




2. What factors are acting to maintain the 












status quo? 












(RESTRAINING FORCES) 












3. What are the individuals' neecs and aspirations? 












(INTERNAL FORCES) 












4. What pressures are exerted on the individual by 












the work context? 












(INDUCED FORCES) 












Rating Scale: 4 - high to 1 = Low TOTALSCORES 












OVERALL RATING 













Figure 41 --Transfer of Learning: Field Analysis Worksheet 
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A second method by which to assess transfer of learning 
issues is to consider each of the key factors in the organization that 
affect human performance. The following worksheet was developed 
based on Sredl and Rothwoll's summary of these factors from the 
literature. Questions to assist the reviewer in considering the 
performance factors concerning external environment, 
organizational culture, organizational structure, organizational 
processes, dominant power coalitions, supervisors, reward systems, 
trainees' roles, work groups, friendship groups, training groups, and 
the individual are given. To work through the worksheet again 
requires listing the issues present under each factor, then assessing 
the program against each factor using the four-point rating scale. 
The programs are assigned an overall rating at the end of the 
process. 
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Program 
Date 



TRANSFER OF LEARNING 
ASSESSMENT WORKSHEET 

Prepared by 



Page 1 of 

I PROGRAM 



1 . EXTERNAL ENVIRONMENT 



What factors in the external environment support tix^ 
transfer of learning? 



2. CRGANIZATONAL CULTURE 

What factors in the organizatioruil culture support the 
application of learning? How consistent is the program with 
the historical beliefs and values of the culture? 



3. ORGANIZATIONAL STRUCTURE 

How/to what extant do factors in the organizational 
structure influence the transfer of learning? 

Division of Labor 
Authority 

Departmentalization 
Span of Control 



4. ORGANIZATIONAL PROCESSES 

What chacteristics of the organizational processes support 
the transfer of learning? 

Decision-making methods 

Communication Practices 

SodalizatJon 



NOTE: Rating System: 4 - Highest to 1 » Lowest 



Totals 
pagol 



i. 



Figure 42 --Transfer of Learning Assessment Worksheet 
(Page 1) 
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Program 
Date 



TRANSFER OF LEARNING 
ASSESSMENT WORKSHEET 

Prepared by 

Page 2 of 

I program" 



5. DOMINANT COALITIONS 

Are the dominant members of various coalitions willing to 
support the training? To what extent will they take actions 
consistent with the training? 



6. SUPERVISORS 

Are the supervisors of the trainees willing to support the 
training? 



7. REWARD SYSTEM 

How will the trainees be rewarded for implementing the 
training? Can explicit rewards be created? 



8. ROLE 

Is the training consistent with the trainees* expectations 
and perceptions of their roles? 



Is the training consistent with other people's expectations 
and perceptions of the trainees' roles? 



NOTE: Rating System: 4 « Highest to 1 = Lowest 



Totals 
page2 



Figure 43>>Transfer of Learning Assessment Worksheet 
(Page 2) 
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TRANSFER OF LEARNING 
ASSESSMENT WORKSHEET 



nnfp Pngri 


5U 

— 3 


nf 


X 




1 PROGRAM 


1 


2 


3 


4 


9. WORKGROUP 












Is the training consistent with the work group's norms, 
values, structure, and expectations for the individual 
who will apply the training? 












10. FRIENDSHIP GROUP 












Is the training consistent with or at least not in conflict with 
values of the trainees' social groups? 










11. TRAINING GROUPS 












What will the effect of the training group be on the oroaram? 










Could the training group be structured to inaease the 
likelihood of members' interacting and reinforcing the 
app<icatk>n of the training after it is completed? 












12. INDIVIDUAL 












How well does the program take into account the individual's 
ability. motivatk)n level, aspirations, and perceived job role? 












page 3 












page2 












page 1 












TOTAL 










Rating System: 4 = Highest to 1 = Lowest 


RATING 











Figure 44 --Transfer of Learning Assessment Worksheet 
(Page 3) 
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Testina/Measurftment Mn the Program^ 

Another important area to consider in this control dimension 
is how testing and measurement of learning is provided. What 
feedback is given to participants about how well they have mastered 
ihe training? The next worksheet assists the reviewer in evaluating 
the testing and measurement of programs. How the testing will be 
performed is considered in the first question. A list of verbal or 
written test items is given. In some programs, the best method for 
testing performance may be a demonstration of the skills. A second 
set of items regarding such a test by demonstration of skills 
follows. This demonstration can be completed without ratings or 
feedback (unrated practice), with ratings and feedback given 
(instructor rated), with participants' ratings of each other 
(participant rated), or through some other method. The second 
question of this worksheet considers the specific feedback elements 
used with the testing. How is the feedback given to the trainee? 
Can the reviewer suggest a way to improve on the feedback methods 
used in the program? Question three considers the transfer of 
learning issue related to the testing and measurement. How well 
does the testing compare with the actual on-the-job application of 
the training? in question four, the reviewer determines if the 
program objectives are consistent with what is tested and 
measured. Based on a consideration of these questions, an overall 
rating is assigned to each program. (This discussion is based in part 
on Sredl & Rothwell, 1987.) 
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TEST & MEASUREMENT WORKSHEET 

Project Prepared by 

D5te Page I of 2 



lill thA tASf inn inMiiHAO 
rill inv iv9iiny iriviiJuv f 


Program 


1 


2 


0% 

3 


4 


Tft^t itftrnft sii^h tut* 










1 1 IJV/i 






























Fill In tha Blank 

• III HI tilw Biai in 










Matching 










Oral Reaponse 










Projectiva 










Other 










Demonstration: 










Unrated Practice 










Instructor Rated 










Participant Rated 










Other: 










Other Test Approaches: 











2. How will feedtNicIc be given to the trainee during the testing 
or measurement? Is that acceptable or can it be improved? 





How \t fMdback given? 


How can It b« Improved? 


Proarsml; 






Program 2: 






Program 3; 






Program 4: 







Figure 45 -Test and Measurement Assessment Worksheet 
(Page 1) 
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TEST & MEASUREMENT WORKSHEET 

(continued) 

Project Prepared by 

Date . Page 2_ if 2 

3. In what ways will the testing differ from the actual job 
application of the training? Are the differences acceptable 
to the reviewer or do they need to be/can they be improved? 

Program 1: 

Program 2. 
Program 3; 

Program 4: 

4. How well does the testing or measurement match tlie stated 
program and unit objectives? 



Program 1: 
Program 2: 
Program 3: 
Prograrr, 4; 



Rating System: 

4 = Highest to 1 = Lowest 


Prog 


ram: 


1 


2 


3 


4 


Overall Rating: 











£ifliiIg_iS--Test and Measurement Assessment Worksheet (Page 2) 
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Summarizing the Ratings 
After the reviewer rates the programs on each of the criteria 
identified as important to tne decision-making proce!>s, the ratings 
are summarized on the summary worksheet. By multiplying the 
ratings by the weightings the reviewer arrives at a weighted number 
for each program on each selected criterion. These weighted 
numbers are totaled, thus giving an overall number for each program. 
The highest number will represent the best program, provided that 
the proper weightings were assigned to each criterion, the correct 
criteria were chosen, and an accurate evaluation was performed 
when the programs were rated against the cnteria. 
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SUMMARY WORKSHEET 

Project OQte 



Prepared by ^ ^ Page of 



CRITERION 


weight 


Dp 

Ratin 


og ram 

1 

^ wt. - 


Program 

2 

Rating/X wt. * 


Program 

3 

RatinQ/X wt - 


Pr 

Ratin 


ogram 
4 

g/X wt. . 































































































































































































































































































































































































































































































































































































































# 1 


















Totals 




• 2 




• 3 




« 4 





Figure 47- Stjmmary Worksheet 
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CHAPTER FIVE ~ FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS 
The purposes of this study were (a) to determine the criteria 
identified in the literature as important to the process of selecting 
training programs, (b) to create a model to account tor the criteria, 
and (c) to develop an instrument for evaluating programs against the 
criteria. A review of the literature was carried out regarding 
decision-making processes, vendors' training programs, 
instructional development systems, training evaluation, 
instructional design and delivery, and organizational influences on 
training. From this review, criteria were identified and separated 
into four dimensions to create the training program selection 
criteria model and checklist. In addition, an instrument consisting 
of worksheets for each criterion was developed and presented to 
assist the training program buyer in evaluating and comparing 
programs and reaching an overall decision. 

In reviewing the findings of this study, each dimension should 
be compared regarding the criteria found in the literature and the 
criteria that were expected. In doing so, the design and content 
dimensions appear to represent most thoroughly the possible 
criteria. Because the design dimension is based on firmly 
established instructional design systems and because much has been 
written about each aspect of the design process, it was expected 
that this list would be more thorough and complete than other 
aspects of the program selection model. 

In the content dimension, focusing on the comparison of the 
content to the assessed needs, the accuracy of the content, and 
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applicability of the content to the trainees' characteristics and 
trainees' readiness provides a complete listing of aspects to be 
considered. To expand the content dimension, a worksheet might be 
developed to determine the costs of including unnecessary content 
or the costs of not including required conteni. The subjeci matter 
review worksheet might be redesigned for several different 
situations, for example, it might be used by a subject matter expert 
to review technical training, or it might ba used by a nontechnical 
manag er to review nontechnical programs. These worksheets could 
be designed to focus on the materials, the program, or the materials 
and the program. 

The context and control dimensions do not represent as 
thorough a listing of possible criteria as the other two dimensions. 
The criteria contained in the control dimension should be expanded 
to contain a cost-benefit analysis criterion. The context dimension 
appears to be the least complete. A worksheet could be developed to 
assist the reviewer in selecting a vendor-supplied instructor. 

The review of decision-making and buying processes showed 
the importance of the factors external to the program. The 
importance that politics plays in the purchasing process would 
hardly be disputed, but it was not mentioned in the literature. The 
criterion cultural fit may have included too many different aspects 
of the decision-making process to be used as a single item. One 
aspect of cultural fit that was not considered was how well the 
program fits into thf» existing curriculum of the buyer's 
organization. 
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Dividing and assigning the criteria into the four dimensions 
may be arbitrary. For example, trainee characteristics could have 
been considered as part cf the design dimension, even though they 
are considered as part of the content dimension. 

An alternative means of mapping the criteria may be a three- 
part model based on criteria directly related to the program, to the 
implementation of the program, and to the organizational 
considerations. More work with people who would actually use this 
model could be done to determine the best presentation of the 
criteria. 

The focus of this paper was on the purchasing of a program 
within an organization. The assumptions of the study and the 
literature considered have led to the development of the various 
criterir. Any change or investigation providing ,-nore concrete 
insight into tho assiimptions would be helpful. The investigator 
assumed that there would be either one person or a team of a few 
people, probably '^ the human resources or training departmeiit of an 
organization, who would be responsible for finding and choosing a 
program. A survey to identify exactly who is involved in selecting 
the training program would be helpful. Determining what criteria 
and information are important to actual purchases of training may 
represent a more valid way to determine present practices. 
Questions could then be directed to the consideration of what 
criteria are more important to some decision makers than to others. 
Certain types of programs may require the consideration of different 
criteria, or the criteria may take on different importance ratings. 
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The docision-making process is largely directed by criteria 
that are important to the primary stakeholders in the decision 
outcome. In future research, perhaps more emphasis could be placed 
on identifying key stakeholders and determining criteria that are 
important to them. The question of the relevance and usefulness of 
the evaluation results *o decision-makers represents another area 
of inquiry to be considered in greater depth for future research and 
studies. Indeed, the major issue with program evaluation may not be 
Whal to evaluate but how to ensure that the evaluation data are used 
by the decision makers. As concluded in one study, "the recent 
literature is unanimous in ar^nouncing the general failure of 
evaluation to affect decision-making in a significant way" (Wholey, 
Scanlon, Duffy. Fukumotu, & Vogt, 1970, as cited in Patton. 1986, p. 
23). 

This study brings together some of the initial perspectives of 
the literature concerning decisions to buy vendor-supplied training 
programs. Much work remains concerning the decision-making 
processes used in selecting vendor-supplied training programs. 

Concluding Comments 

The implications this investigation can have for the 
buyer/seller relationship are considerable. If the majority of 
training buyers used the tools described here, the following might 
result: 

1 . Training programs would be bought and sold because of 
their merit and not because of personalities and politics. 
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2. Vendors would have more difficulty providing 
undocumented results. Many vendors would either have 
to redesign and document their programs or go out of 
business. 

3. Flaws In training programs would be much more evident 
to the buyer earlier In the purchase process. Many flaws 
would be Identified even before the pilot testing of the 
program, thus saving money, time, and expenses. Indeed, 
the decision to pilot-test a program may come too late in 
the orocess to reverse any but the poorest of purchasing 
decisions because of outside pressures to follow through 
with the program. 

4. There may be a drop in training expenditures. Increased 
knowledge on the part of the buyers would put them in a 
better position to negotiate. 

5. The organization ma^y perceive the training function in a 
different light. Because the evaluation process 
discussed here requires the training program purchaser 
to consider the program from many different vantage 
points within the organization, he or she may become 
more integrally associated witn the strategic and 
tactical aspects of organizational planning. 

6. Training decisions would be well documented for later 
evaluation and consideration. Decision makers would be 
able to see both the strengths and the weaknesses in the 
purchasing process, and, subsequently, to apply that 
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learning to other situations. The purchaser would 
develop sharpened critical capabilities (R jers & Voipe, 
1984). 

The buying/selling relationship between training 
professionals and vendors would be redefined. How 
information is shared between the two parties, the 
increased level of trust required to make the 
relationship more effective, and the increased emphasis 
on longer term, continuous interaction are just three 
aspects of the buyer/seller relationship that would 
change. 

Training professionals world be held more accountable 
for their decisions and for determining the impact that 
programs will have on the organization. 
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